Master Server and Analytic Server
From Array Suite Wiki
Master and Analytic Server
Overview
The design of Master and Analytic server setup is to help companies manage distributed computing and data storage across multiple sites (such as US and EU sites). The master server served as central server for management and analytic servers are running near local storage to run extensive analysis.
Roles for Master and Analytic Server
The following list shows some basic difference between Master Server and Analytic Server.
Master Server | Analytic Server |
Management of users, samples, pipelines, shared views/genome browsers and published projects | Server Project Management (individual user's projects) |
Connects directly to a database | Server for server-based analysis |
Configured using ArrayServer.cfg file | Each server is configured by AnalyticServer.cfg file |
Setting up User and ServerProject Folders
Assuming the following setup: one master server called Master and three analytic servers called ServerA, ServerB and ServerC, respectively. Three analytic machines are hosted on three machines and using network storage nearby. A user will login to one of the Analytic servers, but would like to have access to files in other servers. To support this capability, ArrayServer builds a virtual folder structure under each Analytic server's BaseDirectory (in the AnalyticServer.cfg) and defines the physical user and server project folder path in the [AnalyticServer] option in the ArrayServer.cfg. Each BaseDirectory specified in the analytic server contains the shortcuts to each server's "real data" folder.
Once a user logs into the master server, the user will have a choice to go to one analytic server. No matter which analytic server is selected login, the user can still see user folder files in the other two analytic servers.
Syntax for Analytic Server Folder mapping
Folder mapping should be configured in master server's ArrayServer.cfg file. For each analytic server, one line should be added in master server's ArrayServer.cfg file that contains the following seven fields: ServerUrl|ServerName|WindowsUsersDirectory|LinuxUsersDirectory|WindowsServerProjectsDirectory|LinuxServerProjectsDirectory|ServerDescription
The ServerName field cannot have spaces or special characters (i.e. characters not allowed in filenames).
Here are examples for Arrayserver.cfg Analytic server configuration used for three scenarios
Example setup with Windows machines hosting all analytic servers
Use UNC path so that every server machine can access/mount these folders with the same path.
[AnalyticServers] tcp://uswest.omicsoft.com:8065/RemoteServer|ServerA|\\useast\ServerA_Users|NoLinuxUsers|\\useast\ServerA_ServerProjects|NoLinuxServerprojects|Server in US West tcp://useast.omicsoft.com:8065/RemoteServer|ServerB|\\uswest\ServerB_Users|NoLinuxUsers|\\useast\ServerB_ServerProjects|NoLinuxServerprojects|Server in US East tcp://euwest.omicsoft.com:8065/RemoteServer|ServerC|\\euwest\ServerC_Users|NoLinuxUsers|\\eueast\ServerC_ServerProjects|NoLinuxServerprojects|Server in EU
Example setup with Linux machines hosting all analytic servers
Server admin will first mount all three network drives in each machine using the mount path so that every server machine can access files with the same path.
[AnalyticServers] tcp://uswest.omicsoft.com:8065/RemoteServer|ServerA|NoWindowsUsers|//mnt/uswest/ServerA_Users|NoWindowsServerProjects|//mnt/uswest/ServerA_ServerProjects|Server in US West tcp://useast.omicsoft.com:8065/RemoteServer|ServerB|NoWindowsUsers|//mnt/useast/ServerB_Users|NoWindowsServerProjects|//mnt/useast/ServerB_ServerProjects|Server in US East tcp://euwest.omicsoft.com:8065/RemoteServer|ServerC|NoWindowsUsers|//mnt/euwest/ServerC_Users|NoWindowsServerProjects|//mnt/euwest/ServerC_ServerProjects|Server in EU
Example setup with both Linux and Windows machines (mixed)
Note: //mnt/uswest/ServerA_Users (mounted folder in Linux) and \\uswest\ServerA_Users (UNC path) should point to the same folder.
[AnalyticServers] tcp://uswest.omicsoft.com:8065/RemoteServer|ServerA|\\uswest\ServerA_Users|//mnt/uswest/ServerA_Users|\\uswest\ServerA_ServerProjects|//mnt/uswest/ServerA_ServerProjects|Server in US West tcp://useast.omicsoft.com:8065/RemoteServer|ServerB|\\useast\ServerB_Users|//mnt/useast/ServerB_Users|\\useast\ServerB_ServerProjects|//mnt/useast/ServerB_ServerProjects|Server in US East tcp://euwest.omicsoft.com:8065/RemoteServer|ServerC|\\euwest\ServerC_Users|//mnt/euwest/ServerC_Users|\\eueast\ServerC_ServerProjects|//mnt/euwest/ServerC_ServerProjects|Server in EU
User/ServerProjects folders in BaseDirectory
Note, admin has setup BaseDirectory for each Analytic Server. Admin should be able to find ServerA, ServerB, and ServerC folder in BaseDirectory/FtpRoot/Users/xxx/
. It should be empty if you check them outside of ArrayStudio. These are folders setup in FTP as virtual folder mapping to the real folder setup in [AnalyticServers]. For example, in ArrayStudio, the folder /Users/gary/ServerA/Test
is actually pointing to \\useast\ServerA_Users\gary\Test
. Folder mapping is handled by ArrayServer.
Configure Cloud home directory as home directory
Both Master and Analytic servers can have their own AWS S3 as cloud home directories. This can be configured from Master .cfg and Analytic .cfg.
Configuration example
Cloud folder mapping configuration for Master.cfg
[CloudFolder] PrivateCloudLandsFolder=/east1.singlecell.omicsoft/SingleCell CloudLandProjects=/east1.devcluj.omicsoft/TudorT/Projects CloudFolderQaQC=/east.geneticsland.pc.omicsoft/CloudFolderQaQC/RusA ToniFolder=/east.geneticsland.pc.omicsoft/GaleaA MasterHome=/east.geneticsland.pc.omicsoft/GaleaA/ArrayServer_Master
Cloud folder mapping configuration for Analytic.cfg
[CloudFolder] ToniFolder=/east2.geneticsland.pc.omicsoft/GaleaA AnalyticsHome=/east2.geneticsland.pc.omicsoft/GaleaA/ArrayServer_Analytics CloudFolderQaQC=/east.geneticsland.pc.omicsoft/CloudFolderQaQC/RusA
- ServerFiles list the Master home when connected to Master Server and list the Analytic Home when connected to Analytic Server
- Manage Cloud Files actions open the CloudHomeDirectory for each server (e.g. /MasterHome when connected to Master Server and /AnalyticsHome when connected to Analytic Server)
Essential options for master and analytic server configure file
OmicSoft Server v 11.6
For the master server's Arrayserver.cfg, an essential option is:
MasterServerUrl=tcp://192.168.1.129:8065
As the master server can also work as an analytic server, there should be another line:
AnalyticServerName=A1
For A2's AnalyticServer.cfg, the Option section needs to contain at least the following three options:
MasterServerUrl=tcp://192.168.1.129:8065
AnalyticServerName=A2
IsAnalyticServer=True
Once the configure files have been set up, both Maser server machine and Analytic server machine should open the port in firewall, which has been defined in the configure files.
Additional mapped folders can be included in the master server's Arrayserver.cfg file, in the [Folder] section.
The basic syntax is
MappedFolderName=\\Windows\Path\To\Folder | /Linux/Path/To/Folder | AnalyticServerName
It may look like this:
[Folder] ShareFolder=\\192.168.1.1\Shared_Folder | /SharedFolder | A1
You need to provide directories for both Windows (\\192.168.1.1\Shared_Folder) and Linux (/SharedFolder). This ensures that, no matter which system is installed in the analytic server, it will find the path to connect the folder specified in the master server's ArrayServer.cfg file.
Also read ArrayServer.cfg.