Build AWS Ubuntu AMI for OmicSoft Cloud Computing

From Array Suite Wiki

Jump to: navigation, search

Contents

Build AWS AMI for OmicSoft Cloud Computing

OmicSoft provides default AMIs for cloud computing, based on Ubuntu (this page) or Red Hat. Alternatively, users can build their own AMI then follow the configuration steps below.

Currently, on-demand compute instances to analyze ArrayServer jobs must use an Ubuntu image. If a custom Ubuntu image was created for compute instances, this image can be specified in ArrayServer.cfg in the Ami and AmiSnapshot parameters.

The AMI should be built following the instructions below exactly, including the file paths to software tools.

Pre-built OmicSoft Ubuntu images

Steps here are based on Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-74-generic x86_64) as backbone. Here are final public AMI in three regions:

## East Virginia
# built on 05/2016, considered as PROD and copied to other regions
Ami=ami-4033c32d
AmiSnapshot=snap-fe96391a

# built based on ami-4033c32d in 04/2017; fixed the ulimit issue for root user; only users running GATK may hit the issue without this fix.
Ami=ami-5dbe2a4b
AmiSnapshot=snap-07a2e91f794c73678

# built based on ami-5dbe2a4b in 10/2017; considered as beta test; added basic python, R, and docker; updated OmicsoftUpdater to latest to have more DEV versions
Ami=ami-773de00d
AmiSnapshot=snap-0997e3979b768dde3
## west1 California
Ami=ami-60b5ce00
AmiSnapshot=snap-f85d17de
## west2 Oregon
Ami=ami-6aff060a
AmiSnapshot=snap-c323a739

Configure your own Ubuntu image

Install Required Linux packages

Make sure you have packages installed:

sudo apt-get install -f
sudo apt-get update

sudo apt-get install gcc g++ bison make libtool gettext libgdiplus zlib1g-dev libglib2.0-dev
sudo apt-get install libpng-dev libjpeg-dev libtiff5-dev libgif-dev libgs-dev libexif-dev libX11-dev fontconfig
[back to top]

Install Mono 2.10.9

Tips.pngFor Linux kernels prior to 3.10.0-327 we recommended to use mono 2.10.9 in Linux. For newer kernels, mono4.0.4 is required.
Install mono2
  • Download Mono 2.10.9
    cd /opt
    wget -c  http://origin-download.mono-project.com/sources/mono/mono-2.10.9.tar.bz2

The bz2 file can be saved to a temporary location, e.g. ~/temp/

  • Extract and modify certificate if necessary
    tar  jxvf   mono-2.10.9.tar.bz2
    cd  /opt/mono-2.10.9

For mono 2.10.9, it is recommended to modify X509Certificate to the latest standard.

  • Compile and install. On the command line, type
    cd  /opt/mono-2.10.9
    ./configure  --prefix=/opt/mono-2.10.9 --with-large-heap=yes
    make
    make  install

Note:

  • The location of mono installed is set by the option "--prefix" in the configure step, which can be changed to another location.
  • The option --with-large-heap=yes is to enable support for GC heaps larger than 3gb, which is required for NGS alignment, as well as some Array Server functions)

Double check mono installation and version

ls /opt/mono-2.10.9/bin/mono* -all
/opt/mono-2.10.9/bin/mono --version
/opt/mono-2.10.9/bin/mono-sgen --version
[back to top]

use sudo in ubuntu.

Install Mono 4.0.4

Install Mono 4.0.4 is optional; just in case some proc requires newer mono. Mono 4.0.4 contains latest certificate standard.

Install Mono 4.0.4
  • Download Mono 4.0.4
    wget -c http://download.mono-project.com/sources/mono/mono-4.0.4.1.tar.bz2

The bz2 file can be saved to a temporary location, e.g. ~/temp/

  • Compile and install. On the command line, type
    tar jxvf mono-4.0.4.1.tar.bz2
    cd  mono-4.0.4
    ./configure  --prefix=/opt/mono-4.0.4 --with-large-heap=yes
    make
    make  install

Note:

  • The location of mono installed is set by the option "--prefix" in the configure step, which can be changed to another location.
  • The option --with-large-heap=yes is to enable support for GC heaps larger than 3gb, which is required for NGS alignment, as well as some Array Server functions)

Double check mono installation and version

ls /opt/mono-4.0.4/bin/mono* -all
/opt/mono-4.0.4/bin/mono --version
/opt/mono-4.0.4/bin/mono-sgen --version
[back to top]

use sudo in ubuntu.

Install libgdiplus

sudo apt-get install libgdiplus

Sqlite

Sqlite can be downloaded from: sqlite download. The tar.gz file is recommended for installation.

wget -c http://www.sqlite.org/sqlite-autoconf-3071401.tar.gz
tar zxvf sqlite-autoconf-3071401.tar.gz
sudo mv sqlite-autoconf-3071401 sqlite
cd sqlite
./configure --prefix=/opt/sqlite
make
make install

sqlite3  --version
[back to top]

use sudo in ubuntu.

Set ulimit

Make sure the ulimit for "max user processes" and "open files" are set to the max value: 65536. You can check the values by typing: ulimit -a.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515184
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Modify ulimit in two config files following ulimit setup wiki.


Make sure ulimit for root (create user root for ubuntu) has the same ulimit.

[back to top]


Install Oshell

cd /opt
sudo mkdir oshell
cd oshell
sudo wget -c  http://omicsoft.com/software_update/OmicsoftUpdater.exe
sudo touch oshell.exe
sudo /opt/mono-2.10.9/bin/mono /opt/oshell/OmicsoftUpdater.exe
sudo /opt/mono-2.10.9/bin/mono /opt/oshell/oshell.exe --version
[back to top]


Install AWS CLI

sudo apt-get install python-pip
sudo pip install awscli
[back to top]


Optional installation for dev features

Multiple Python versions

sudo apt-get install python3.6
sudo apt-get install python3-pip

Then different version of python can be called:

ubuntu@ip-172-31-12-154:/opt/oshell$ python -V
Python 2.7.6
ubuntu@ip-172-31-12-154:/opt/oshell$ python3.6 -V
Python 3.6.3
ubuntu@ip-172-31-12-154:/opt/oshell$ python3.6
Python 3.6.3 (default, Oct  6 2017, 00:00:00)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

R basics and a few useful packages

sudo apt-get install r-base
sudo R

Then install packages

source("http://bioconductor.org/biocLite.R")
install.packages("samr"); 
install.packages("Rtsne");
biocLite("pvca")
biocLite("Biobase")
biocLite("lme4")
biocLite("vsn")
biocLite("limma")
biocLite("edgeR")
biocLite("impute")
biocLite("lumi")

Install Docker

Following https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-using-the-repository

For Ubuntu 14.04, we did

sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce

Test

sudo docker run hello-world

ubuntu@ip-172-31-12-154:~$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
5b0f327be733: Pull complete
Digest: sha256:07d5f7800dfe37b8c2196c7b1c524c33808ce2e0f74e7aa00e603295ca9a0972
Status: Downloaded newer image for hello-world:latest 

Hello from Docker!
This message shows that your installation appears to be working correctly.

Related Articles

[back to top]

EnvelopeLarge2.png