Build AWS Ubuntu AMI for OmicSoft Cloud Computing

From Array Suite Wiki

Jump to: navigation, search

Warning.png WARNING: This page is deprecated. Please visit Install OmicSoft v12 on Ubuntu20 for Omicsoft Cloud Computing for updated information on newer versions.


Contents

AWS AMI for OmicSoft Cloud Computing

OmicSoft Cloud uses customized Amazon Machine Images (AMIs) as the template for creating compute instances for running cloud-based analyses. These AMIs require a few key components, including the Oshell engine, packages for transferring data to and from AWS S3 storage, and Docker.

OmicSoft provides default AMIs for cloud computing, based on Ubuntu. Alternatively, users can build their own AMI then follow the configuration steps below.

On-demand compute instances to analyze ArrayServer jobs (i.e. AMIs used for OmicSoft Cloud analysis) must use an Ubuntu image. We recommend explicitly specifying the preferred AMI from the list below, in ArrayServer.cfg in the Ami and AmiSnapshot parameters, or by building your own Ubuntu AMI with the configuration steps outlined below.

If you build your own AMI, the AMI should be built following the instructions below exactly, including the file paths to software tools. Please be aware the OmicSoft support can only provide limited advice on troubleshooting custom AMIs, because of the large number of configuration options that can impact performance.

Pre-built OmicSoft Ubuntu images

Pre-built OmicSoft Server Ubuntu20 images

OmicSoft recommends using our pre-configured Ubuntu20 AMI for installing OmicSoft Server in the AWS environment. Find the AMI for your region here.

Pre-built cloud analysis AMIs

For both onsite and cloud-based installations of OmicSoft server, cloud NGS analysis add-on works best with mono6-based AMIs for Cloud analysis: OmicSoft Cloud Analysis AMIs.

Legacy information: Configure your own Ubuntu 14 AWS image

Previous releases of OmicSoft Suite were compatible with pre-built mono4-based AMIs: Pre-built_OmicSoft_Ubuntu_AMIs#Mono4-based_AMIs_.28previous_generation.29

We recommend using a pre-built AMI (above) or building an analysis AMI on a modern OS such as Ubuntu20.

The following information is for reference purposes.

Install Required Linux packages

Make sure you have packages installed:

sudo apt-get install -f
sudo apt-get update

sudo apt-get install gcc g++ bison make libtool gettext libgdiplus zlib1g-dev libglib2.0-dev
sudo apt-get install libpng-dev libjpeg-dev libtiff5-dev libgif-dev libgs-dev libexif-dev libX11-dev fontconfig
[back to top]

Install Mono

Install Mono 4.0.4

  • Download Mono 4.0.4
    wget -c http://download.mono-project.com/sources/mono/mono-4.0.4.1.tar.bz2

The bz2 file can be saved to a temporary location, e.g. ~/temp/

  • Compile and install. On the command line, type
    tar jxvf mono-4.0.4.1.tar.bz2
    cd  mono-4.0.4
    ./configure  --prefix=/opt/mono-4.0.4 --with-large-heap=yes
    make
    make  install

Note:

  • The location of mono installed is set by the option "--prefix" in the configure step, which can be changed to another location.
  • The option --with-large-heap=yes is to enable support for GC heaps larger than 3gb, which is required for NGS alignment, as well as some Array Server functions)

Double check mono installation and version

ls /opt/mono-4.0.4/bin/mono* -all
/opt/mono-4.0.4/bin/mono --version
/opt/mono-4.0.4/bin/mono-sgen --version


Install Mono 4.8.1

  • Download Mono 4.8.1
    wget -c http://download.mono-project.com/sources/mono/mono-4.8.1.0.tar.bz2

The bz2 file can be saved to a temporary location, e.g. ~/temp/

  • Compile and install. On the command line, type
    tar jxvf mono-4.8.1.0.tar.bz2
    cd  mono-4.8.1
    ./configure  --prefix=/opt/mono-4.8.1 --with-large-heap=yes
    make
    make  install

Note:

  • The location of mono installed is set by the option "--prefix" in the configure step, which can be changed to another location.
  • The option --with-large-heap=yes is to enable support for GC heaps larger than 3gb, which is required for NGS alignment, as well as some Array Server functions)
  • if you get an error like "processes.c:1672:12: error: implicit declaration of function ‘makedev’ [-Werror=implicit-function-declaration]":
add
#include <sys/sysmacros.h>
to mono/io-layer/processes.c

Double check mono installation and version

ls /opt/mono-4.8.1/bin/mono* -all
/opt/mono-4.8.1/bin/mono --version
/opt/mono-4.8.1/bin/mono-sgen --version


To review previous instructions on installing mono-2.10.9 (previous version), visit Install mono 2.10.9

[back to top]

use sudo in ubuntu.


Install libgdiplus

sudo apt-get install libgdiplus

Sqlite

Install SQLite v3.7.14.1 (2012-10-04) by building it from source:

# Download SQLite source archive:
$ cd /opt
$ sudo wget -c http://www.sqlite.org/sqlite-autoconf-3071401.tar.gz
$ sudo tar zxvf sqlite-autoconf-3071401.tar.gz
$ sudo mv sqlite-autoconf-3071401 sqlite
$ cd sqlite

# Build and install:
$ sudo ./configure --prefix=/opt/sqlite
$ sudo make
$ sudo make install

# Check SQLite was installed successfully:
$ sudo ./sqlite3 --version

# Clean up:
$ sudo rm /opt/sqlite-autoconf-3071401.tar.gz
$ cd ~
[back to top]

use sudo in ubuntu.

Set ulimit

Make sure the ulimit for "max user processes" and "open files" are set to the max value: 65536. You can check the values by typing: ulimit -a.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515184
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Modify ulimit in two config files following ulimit setup wiki.


Make sure ulimit for root (create user root for ubuntu) has the same ulimit.

[back to top]


Install Oshell

cd /opt
sudo mkdir oshell
cd oshell
sudo wget -c  http://omicsoft.com/software_update/OmicsoftUpdater.exe
sudo touch oshell.exe
sudo /opt/mono-4.8.1/bin/mono /opt/oshell/OmicsoftUpdater.exe
sudo /opt/mono-4.8.1/bin/mono /opt/oshell/oshell.exe --version
[back to top]


Install AWS CLI

sudo apt-get install python-pip
sudo pip install awscli
[back to top]


Optional installation for dev features

Multiple Python versions

sudo apt-get install python3.6
sudo apt-get install python3-pip

Then different version of python can be called:

ubuntu@ip-172-31-12-154:/opt/oshell$ python -V
Python 2.7.6
ubuntu@ip-172-31-12-154:/opt/oshell$ python3.6 -V
Python 3.6.3
ubuntu@ip-172-31-12-154:/opt/oshell$ python3.6
Python 3.6.3 (default, Oct  6 2017, 00:00:00)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

R basics and a few useful packages

sudo apt-get install r-base
sudo R

Then install packages

source("http://bioconductor.org/biocLite.R")
install.packages("samr"); 
install.packages("Rtsne");
biocLite("pvca")
biocLite("Biobase")
biocLite("lme4")
biocLite("vsn")
biocLite("limma")
biocLite("edgeR")
biocLite("impute")
biocLite("lumi")

Install Docker

Following https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-using-the-repository

For Ubuntu 14.04, we did

sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce

Test

sudo docker run hello-world

ubuntu@ip-172-31-12-154:~$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
5b0f327be733: Pull complete
Digest: sha256:07d5f7800dfe37b8c2196c7b1c524c33808ce2e0f74e7aa00e603295ca9a0972
Status: Downloaded newer image for hello-world:latest 

Hello from Docker!
This message shows that your installation appears to be working correctly.

Related Articles

[back to top]

EnvelopeLarge2.png