Steps to build AWS ArrayServer with cloud plugin

From Array Suite Wiki

Jump to: navigation, search

Contents

Setup ArrayServer in AWS with OmicSoft Cloud plugin

Overview

Company can setup AWS integration in two ways:

  1. ArrayServer is run on local server, side by side with internal HPC, with mapped Cloud S3 folders to perform cloud-based analysis
  2. All-In-AWS cloud setup, where all machines (including the ArrayServer machine) are in AWS

Steps in this article will focus on setup All-In-AWS:

AWSFigure02.png

[back to top]


Steps

Follow these steps from top-to-bottom to set up your cloud-based ArrayServer machine, or on-demand credentials for Studio-based Cloud analysis.

Set AWS Credentials

Set correct region

Be sure to create S3 buckets, EC2 instances and key pairs in an AWS region close to your company.

Create key pair

  • Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/
  • In the navigation pane, under NETWORK & SECURITY, choose Key Pairs.
    • Example Key pair name: Omicsoft.Launching
    • The browser should automatically prompt to download/save the private key file "OmicsoftLaunching.pem"

Create VPC

  • Open the Amazon VPC console at https://console.aws.amazon.com/vpc/
  • Choose desired VPC configuration. Create a simple VPC with a single public subnet by using the "VPC Wizard".
  • Remember the subnet ID to be used in AWS config

Create S3 Bucket

Create a storage location for user data that will be processed by OmicSoft. OmicSoft functions that use data stored in this S3 bucket and output to this S3 bucket will be run on spawned-off EC2 instances, allowing dynamic parallelization of data processing.

  • Sign into the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3
  • Click Create Bucket.
  • Create a Bucket Name, such as east.companyname.omicsoft; choose same region as the ArrayServer machine.
  • Remember to add access to the S3 bucket, and its folders, to the arrayserver policy.

Create IAM Users

  • Sign in to the Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/
  • In the navigation pane, choose Users and then choose Create New Users
    • Example User name: arrayserver
    • "Access Key Id" and "Secret Access Key" will be generated for the new user. Be sure to make a note of these!

Create Policy

  • Sign in to the Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/
  • In the navigation column on the left, choose Policies.
  • At the top of the page, choose Create Policy.
  • In "Create Your Own Policy", create policy name "arrayserver", and use the example policy to specify all of the required permissions.

Create Role

  • Sign in to the Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/
  • In the navigation pane, choose Roles, and then choose Create New Role.
  • Give the Role name "arrayserver"
  • Select "Amazon EC2"
  • Attach "arrayserver" policy created above

Attach Policy To User

  • Select "arrayserver" user created above
  • Under "Permissions" tab, attach policy

Launch and Configure Cloud-based ArrayServer

Launch an instance into subnet

  • Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/
  • On the dashboard, choose Launch Instance.
    • As a convenience, we have generated Red Hat and Ubuntu images, which already have required packages installed. Alternatively, you can select a minimal Linux or Windows image and follow instructions on Getting Started.
      • The OmicSoft Red Hat and Ubuntu AMI can be found by choosing "Community AMIs", then searching for the AMI names mentioned on the relevant pages.
  • Select appropriate instance type, e.g. "r5.xlarge".
    • r5.xlarge will have enough attached memory to load all OmicLands from DiseaseLand and OncoLand.
    • If focusing on analysis, a t2.xlarge (4 CPU, 16GB RAM) may be optimal
  • Select appropriate VPC, Subnet.
    • Enable "Auto-assign Public IP"
    • Enable *"Protect against accidental termination"*
  • Attach EBS storage, for example 500GB "General Purpose SSD"
    • Uncheck *"Delete on Termination"*
  • Create tag with key "Name" and Value "ArrayServer"
  • Instance Security settings
    • This is highly dependent on user needs, but these are some common settings
      • TCP Ports 8064-8066 (for core Array Studio-ArrayServer connections, corresponding to ArrayServer.cfg parameters Port,Port2, and Port3)
      • TCP Port 22 for SSH to the VM
      • TCP ports 60660-60680 (for multi-threaded FTP connections, corresponding to ArrayServer.cfg parameters DataPortBegin and DataPortEnd)
    • Generally, the IP ranges will be restricted to a predefined set of IPs from the company network.
  • Launch with the "Omicsoft.Launching" key pair, or a separate pair
  • Make a note of the public IP address.

Access AWS Instance

  • Read: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html
  • Start PuTTYgen (for example, from the Start menu, click All Programs > PuTTY > PuTTYgen).
  • Under Type of key to generate, select SSH-2 RSA
  • Click Load. By default, PuTTYgen displays only files with the extension .ppk. To locate your .pem file, select the option to display files of all types.
  • Click Save private key to save the key in the format that PuTTY can use. PuTTYgen displays a warning about saving the key without a passphrase. Click Yes.
  • Key passphrase: "omicsoft"
  • Connect with Putty
  • In the Host Name box, enter user_name@public_dns_name. Be sure to specify the appropriate user name for your AMI.
    • For an Amazon Linux AMI, the user name is ec2-user.
    • For a RHEL5 AMI, the user name is either root or ec2-user.
    • For an Ubuntu AMI, the user name is ubuntu.
    • Here I use ec2-user@xxxxxxx.compute-1.amazonaws.com
  • Category -> Connection -> SSH -> Auth, browse and load the .ppk file.

Mount EBS Storage

sudo file -s /dev/xvdb
  • If the output of the previous command shows simply data for the device, then there is no file system on the device and you need to create one.
  • Format the EBS drive if necessary
sudo mkfs -t ext4 /dev/xvdb
  • mount the EBS drive
sudo mkdir /arrayserver
sudo mount /dev/xvdb /arrayserver
  • To mount the EBS drive on every system reboot. Find the device uuid
sudo file -s /dev/xvdb
    • Add the following entry to /etc/fstab to mount an ext4 file system at mount point /arrayserver
UUID=90878434-dad9-4723-829a-xxxxxx /arrayserver	ext4	defaults,nofail	0	2
    • To check wheter the entry will work as expected:
sudo mount -a

Set Up ArrayServer

If an OmicSoft Red Hat or Ubuntu AMI was used, all Linux packages, including mono, libgdiplus, and Sqlite, have been installed in the AMI in /opt directory. Otherwise, please follow the steps in Install ArrayServer dependencies to install required packages.

To set up the ArrayServer application, follow steps in Install ArrayServer to get Omicsoft Updater and ArrayServer with configuration template.

Open Ports

ArrayServer uses certain ports for communications between client and server. These ports must be opened.

Start ArrayServer and get connected

Follow Typical way to update/restart ArrayServer to update/start/stop ArrayServer.

Password protect the ArrayServer Access

Set the following in ArrayServer.cfg

AutoCreateNewUsers=True
UserAuthorization=False

Then enforce CFRPart11 (optional)

Related Articles

[back to top]

EnvelopeLarge2.png