ArrayServer Configuration with Cloud

From Array Suite Wiki

Jump to: navigation, search


Admin has to add two sections to ArrayServer.cfg file to enable Cloud integration to ArrayServer. It will expand ArrayServer with unlimited storage and computing in cloud.

If the Master-Analytic server is used, [Cloud] section has to be specified for each analytic server AnalyticServer.cfg separately. The [Cloud] options can be different for each analytic or master server.
However, in order for all master or analytic servers to be able to access the same cloud folders, the same [CloudFolder] definitions need to be added in every AnalyticServer.cfg and ArrayServer.cfg, including the scenario with multiple S3 accounts.

Add [Cloud] section to each AnalyticServer.cfg.
Add the same [CloudFolder] sections to each AnalyticServer.cfg and ArrayServer.cfg.

Contents

Cloud

[Cloud] section defines Cloud Preferences.
Example using Amazon Cloud

[Cloud]
Provider=Amazon
Region=us-east-1
AccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxx
SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
UseHttp=False
OmicsoftCloudDirectory=s3://east.dev.omicsoft/ArrayServerOmicsoftHome
MaxInstanceCount=5
MaxInstanceCountPerJob=3
UseReducedRedundancy=False
EnableDataEncryption=False
DefaultCloudJobNumber=50
InstanceProfileArn=arn:aws:iam::123xxxxxxxx4:instance-profile/project/omicsoft-xxxxxxxxxxxx
SubnetID=subnet-348b0743
EnableAWSSpot=False

CloudFolder

[CloudFolder] section defines folder mapping to ArrayServer file system.

[CloudFolder]
GaryCloudFolder=/omicsoft.test.gary/gary
SGECloudFolder=/east.dev.omicsoft/SGECloudFolder


For example, GaryCloudFolder and SGECloudFolder will be two folder in ArrayServer file root folder along with your other folder mapping using [Folder] section:

CloudFoldersinArrayServer.png

Multiple S3 Accounts

(available in ArraySuite 2019 R2)

Administrators can configure multiple S3 accounts, each with their own credentials. Each separate S3 account should have its own [CloudFolder] section, along with an AccessKey and SecretKey for that S3 account.


Considerations:

  • this feature only works when UseCli is set to False, in the [Cloud] section
  • any [CloudFolder] section that does not have an AccessKey and SecretKey, will be assumed to belong to the default AWS account, with credentials defined in the [Cloud] section (see SGECloudFolder in the example below)
  • any number of [CloudFolder] sections can be defined, for every separate S3 account
  • any number of folders can be defined within a single [CloudFolder] section
  • all folders defined in a [CloudFolder] section must belong to the same S3 account
  • for each S3 bucket not accessible by the default specified AWS account, an AWS user account should be defined (with Access Key and Secret Key) with an attached policy to give permissions to list buckets, and manipulate objects in those buckets. Specifically, the user account should have a policy that includes the permissions outlined in SID: AllowGroupToSeeBucketListInTheConsole and SID: AllowRootAndHomeListingOfOmicsoftBucket in the Example AWS policy


Example: SGECloudFolder (bucket east.dev.omicsoft) belongs to the AWS root account, while GaryCloudFolder (bucket omicsoft.test.gary) belongs to a different account, with different credentials.

[Cloud]
Provider=Amazon
Region=us-east-1
AccessKey=access_key_root_account
SecretKey=secret_key_root_account
UseHttp=False
OmicsoftCloudDirectory=s3://east.dev.omicsoft/ArrayServerOmicsoftHome
UseCli=False

[CloudFolder]
SGECloudFolder=/east.dev.omicsoft/SGECloudFolder

[CloudFolder]
AccessKey=access_key_additional_account
SecretKey=secret_key_additional_account
GaryCloudFolder=/omicsoft.test.gary/gary

VPC

For ArrayServer configuration with VPC, admin needs additional Cloud options: instance profile and VPC subnet, example:

[Cloud]
Provider=Amazon
Region=us-east-1
AccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxx
SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
UseHttp=False
OmicsoftCloudDirectory=s3://east.dev.omicsoft/ArrayServerOmicsoftHome
MaxInstanceCount=5
MaxInstanceCountPerJob=3
UseReducedRedundancy=False
EnableDataEncryption=True
DefaultCloudJobNumber=50
InstanceProfileArn=arn:aws:iam::123xxxxxxxx4:instance-profile/project/omicsoft-xxxxxxxxxxxx
SubnetID=subnet-348b0743
EnableAWSSpot=False