Master and Analytic Server

Overview

The design of Master and Analytic server setup is to help companies manage distributed computing and data storage across multiple sites (such as US and EU sites). The master server served as central server for management and analytic servers are running near local storage to run extensive analysis.

Roles for Master and Analytic Server

The following list shows some basic difference between Master Server and Analytic Server.

Setting up User and ServerProject Folders

Assuming the following setup: one master server called Master and three analytic servers called ServerA, ServerB and ServerC, respectively. Three analytic machines are hosted on three machines and using network storage nearby. A user will login to one of the Analytic servers, but would like to have access to files in other servers. To support this capability, ArrayServer builds a virtual folder structure under each Analytic server's BaseDirectory (in the AnalyticServer.cfg) and defines the physical user and server project folder path in the [AnalyticServer] option in the ArrayServer.cfg. Each BaseDirectory specified in the analytic server contains the shortcuts to each server's "real data" folder.

Once a user logs into the master server, the user will have a choice to go to one analytic server. No matter which analytic server is selected login, the user can still see user folder files in the other two analytic servers.

Syntax for Analytic Server Folder mapping

Folder mapping should be configured in master server's ArrayServer.cfg file. For each analytic server, one line should be added in master server's ArrayServer.cfg file that contains the following seven fields: ServerUrl|ServerName|WindowsUsersDirectory|LinuxUsersDirectory|WindowsServerProjectsDirectory|LinuxServerProjectsDirectory|ServerDescription

The ServerName field cannot have spaces or special characters (i.e. characters not allowed in filenames).

Here are examples for Arrayserver.cfg Analytic server configuration used for three scenarios

Example setup with Windows machines hosting all analytic servers

Use UNC path so that every server machine can access/mount these folders with the same path.

[AnalyticServers]
tcp://uswest.omicsoft.com:8065/RemoteServer|ServerA|\\useast\ServerA_Users|NoLinuxUsers|\\useast\ServerA_ServerProjects|NoLinuxServerprojects|Server in US West
tcp://useast.omicsoft.com:8065/RemoteServer|ServerB|\\uswest\ServerB_Users|NoLinuxUsers|\\useast\ServerB_ServerProjects|NoLinuxServerprojects|Server in US East
tcp://euwest.omicsoft.com:8065/RemoteServer|ServerC|\\euwest\ServerC_Users|NoLinuxUsers|\\eueast\ServerC_ServerProjects|NoLinuxServerprojects|Server in EU

Example setup with Linux machines hosting all analytic servers

Server admin will first mount all three network drives in each machine using the mount path so that every server machine can access files with the same path.

[AnalyticServers]
tcp://uswest.omicsoft.com:8065/RemoteServer|ServerA|NoWindowsUsers|//mnt/uswest/ServerA_Users|NoWindowsServerProjects|//mnt/uswest/ServerA_ServerProjects|Server in US West
tcp://useast.omicsoft.com:8065/RemoteServer|ServerB|NoWindowsUsers|//mnt/useast/ServerB_Users|NoWindowsServerProjects|//mnt/useast/ServerB_ServerProjects|Server in US East
tcp://euwest.omicsoft.com:8065/RemoteServer|ServerC|NoWindowsUsers|//mnt/euwest/ServerC_Users|NoWindowsServerProjects|//mnt/euwest/ServerC_ServerProjects|Server in EU

Example setup with both Linux and Windows machines (mixed)

Note: //mnt/uswest/ServerA_Users (mounted folder in Linux) and \\uswest\ServerA_Users (UNC path) should point to the same folder.

[AnalyticServers]
tcp://uswest.omicsoft.com:8065/RemoteServer|ServerA|\\uswest\ServerA_Users|//mnt/uswest/ServerA_Users|\\uswest\ServerA_ServerProjects|//mnt/uswest/ServerA_ServerProjects|Server in US West
tcp://useast.omicsoft.com:8065/RemoteServer|ServerB|\\useast\ServerB_Users|//mnt/useast/ServerB_Users|\\useast\ServerB_ServerProjects|//mnt/useast/ServerB_ServerProjects|Server in US East
tcp://euwest.omicsoft.com:8065/RemoteServer|ServerC|\\euwest\ServerC_Users|//mnt/euwest/ServerC_Users|\\eueast\ServerC_ServerProjects|//mnt/euwest/ServerC_ServerProjects|Server in EU

User/ServerProjects folders in BaseDirectory

Note, admin has setup BaseDirectory for each Analytic Server. Admin should be able to find ServerA, ServerB, and ServerC folder in BaseDirectory/FtpRoot/Users/xxx/. It should be empty if you check them outside of ArrayStudio. These are folders setup in FTP as virtual folder mapping to the real folder setup in [AnalyticServers]. For example, in ArrayStudio, the folder /Users/gary/ServerA/Test is actually pointing to \\useast\ServerA_Users\gary\Test. Folder mapping is handled by ArrayServer.

Configure Cloud home directory as home directory

Both Master and Analytic servers can have their own AWS S3 as cloud home directories. This can be configured from Master .cfg and Analytic .cfg.

Configuration example

Cloud folder mapping configuration for Master.cfg

[CloudFolder]
PrivateCloudLandsFolder=/east1.singlecell.omicsoft/SingleCell
CloudLandProjects=/east1.devcluj.omicsoft/TudorT/Projects
CloudFolderQaQC=/east.geneticsland.pc.omicsoft/CloudFolderQaQC/RusA
ToniFolder=/east.geneticsland.pc.omicsoft/GaleaA
MasterHome=/east.geneticsland.pc.omicsoft/GaleaA/ArrayServer_Master

Cloud folder mapping configuration for Analytic.cfg

[CloudFolder]
ToniFolder=/east2.geneticsland.pc.omicsoft/GaleaA
AnalyticsHome=/east2.geneticsland.pc.omicsoft/GaleaA/ArrayServer_Analytics
CloudFolderQaQC=/east.geneticsland.pc.omicsoft/CloudFolderQaQC/RusA

• ServerFiles list the Master home when connected to Master Server and list the Analytic Home when connected to Analytic Server
• Manage Cloud Files actions open the CloudHomeDirectory for each server (e.g. /MasterHome when connected to Master Server and /AnalyticsHome when connected to Analytic Server)

Essential options for master and analytic server configure file

OmicSoft Server v 11.6

For the master server's Arrayserver.cfg, an essential option is:

MasterServerUrl

MasterServerUrl=tcp://192.168.1.129:8065

As the master server can also work as an analytic server, there should be another line:

AnalyticServerName=A1

For A2's AnalyticServer.cfg, the Option section needs to contain at least the following three options:

MasterServerUrl

MasterServerUrl=tcp://192.168.1.129:8065

AnalyticServerName

AnalyticServerName=A2

IsAnalyticServer

IsAnalyticServer=True

Once the configure files have been set up, both Maser server machine and Analytic server machine should open the port in firewall, which has been defined in the configure files.

Shared folder specified in master server

Additional mapped folders can be included in the master server's Arrayserver.cfg file, in the [Folder] section.

The basic syntax is

MappedFolderName=\\Windows\Path\To\Folder | /Linux/Path/To/Folder | AnalyticServerName

If the folder to map is on the master server, AnalyticServerName should be Master

It may look like this:

[Folder]
ShareFolder=\\192.168.1.1\Shared_Folder | /SharedFolder | A1

You need to provide directories for both Windows (\\192.168.1.1\Shared_Folder) and Linux (/SharedFolder). This ensures that, no matter which system is installed in the analytic server, it will find the path to connect the folder specified in the master server's ArrayServer.cfg file.

Also read ArrayServer.cfg.