ArrayServer Configuration with Cloud
From Array Suite Wiki
Admin has to add two sections to ArrayServer.cfg file to enable Cloud integration to ArrayServer. It will expand ArrayServer with unlimited storage and computing in cloud.
If the Master-Analytic server is used, [Cloud] section has to be specified for each analytic server AnalyticServer.cfg separately. The [Cloud] options can be different for each analytic or master server. However, in order for all master or analytic servers to be able to access the same cloud folders, the same [CloudFolder] definitions need to be added in every AnalyticServer.cfg and ArrayServer.cfg, including the scenario with multiple S3 accounts.
Add [Cloud] section to each AnalyticServer.cfg. Add the same [CloudFolder] sections to each AnalyticServer.cfg and ArrayServer.cfg.
Example using Amazon Cloud
[Cloud] section defines Cloud Preferences.
[Cloud] Provider=Amazon Region=us-east-1 AccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxx SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx UseHttp=False OmicsoftCloudDirectory=s3://east.dev.omicsoft/ArrayServerOmicsoftHome MaxInstanceCount=5 MaxInstanceCountPerJob=3 UseReducedRedundancy=False EnableDataEncryption=False DefaultCloudJobNumber=50 InstanceProfileArn=arn:aws:iam::123xxxxxxxx4:instance-profile/project/omicsoft-xxxxxxxxxxxx SubnetID=subnet-348b0743 EnableAWSSpot=True
[CloudFolder] section defines folder mapping to ArrayServer file system.
[CloudFolder]] GaryCloudFolder=/omicsoft.test.gary/gary SGECloudFolder=/east.dev.omicsoft/SGECloudFolder
For example, GaryCloudFolder and SGECloudFolder will be two folder in ArrayServer file root folder along with your other folder mapping using ArrayServer folder mapping and management|[Folder] section]]:
Multiple S3 Accounts
Administrators can configure multiple S3 accounts, each with their own credentials. Each separate S3 account should have its own [CloudFolder] section, along with an AccessKey and SecretKey for that S3 account.
- This feature only works when UseCli is set to False, in the [Cloud] section.
- Any [CloudFolder] section that does not have an AccessKey and SecretKey, will be assumed to belong to the default AWS account, with credentials defined in the [Cloud] section (see SGECloudFolder in the example below)
- any number of [CloudFolder] sections can be defined, for every separate S3 account
- any number of folders can be defined within a single [CloudFolder] section
- all folders defined in a [CloudFolder] section must belong to the same S3 account
- for each S3 bucket not accessible by the default specified AWS account, an AWS user account should be defined (with Access Key and Secret Key) with an attached policy to give permissions to list buckets, and manipulate objects in those buckets. Specifically, the user account should have a policy that includes the permissions outlined in
SID: AllowGroupToSeeBucketListInTheConsole SID: AllowRootAndHomeListingOfOmicsoftBucket
in the Example AWS policy
Example: SGECloudFolder (bucket east.dev.omicsoft) belongs to the AWS root account, while GaryCloudFolder (bucket omicsoft.test.gary) belongs to a different account, with different credentials.
[Cloud] Provider=Amazon Region=us-east-1 AccessKey=access_key_root_account SecretKey=secret_key_root_account UseHttp=False OmicsoftCloudDirectory=s3://east.dev.omicsoft/ArrayServerOmicsoftHome UseCli=False [CloudFolder] SGECloudFolder=/east.dev.omicsoft/SGECloudFolder [CloudFolder] AccessKey=access_key_additional_account SecretKey=secret_key_additional_account GaryCloudFolder=/omicsoft.test.gary/gary
For OmicSoft Server configuration with VPC, admin needs additional Cloud options: instance profile and VPC subnet.
[Cloud] Provider=Amazon Region=us-east-1 AccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxx SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx UseHttp=False OmicsoftCloudDirectory=s3://east.dev.omicsoft/ArrayServerOmicsoftHome MaxInstanceCount=5 MaxInstanceCountPerJob=3 UseReducedRedundancy=False EnableDataEncryption=True DefaultCloudJobNumber=50 InstanceProfileArn=arn:aws:iam::123xxxxxxxx4:instance-profile/project/omicsoft-xxxxxxxxxxxx SubnetID=subnet-348b0743