Enable swap file for memory
From Array Suite Wiki
How to add a swap file on Linux Servers to protect against out-of-memory errors
With increasing sizes of Land databases and analyses, machines that previously would be considered to have ample RAM (32GB) can find all memory used up by large searches, especially with Single Cell Lands. When all memory is used, even temporarily, Linux systems employ an "OOM killer" to stop the mono process running ArrayServer process.
However, it is simple to add a "swap" file to act as a sort of overflow to store some of the data in-memory, reducing the likelihood of an Out-of-memory error and server crash.
The following steps should be performed by the OmicSoft Server administrator or relevant IT team member.
Q. Assuming the customer has the disk space, is there a downside to enabling a swap file? A. Not that we are aware of; this is quite routinely used. When there is not enough memory for all applications in a system without swapspace it will cause failure to allocate memory for new applications requesting new memory pages and this usually results in termination of the program. So having swapspace is usually recommended.
Q. How big should my swap file be? Can I make a swap space smaller than my RAM? (some customers have huge machines with 512GB RAM , so the swap won't be needed anyway) A. Swap space can be of any size. For a 32-64GB machine, 6-8 GB should be a good middle-ground file. See this web page for more guidance.
Q. Does this work works on Ubuntu and CentOS/RHEL? A. This has been tested on all three common Linux OSs without issues, when following the steps below.
Q. Some customers launch OmicSoft Server under Root; others launch under a service account. Does this change the commands at step 3 (below)? A. No, On Ubuntu and CentOS we have launched OmicSoft Server using used the default service account provided by AWS. It should be ok.
Steps to add a Swap File
In this example, we will make a swap file in a subdirectory of disk /dev/nvme1n1, in the folder /opt/ArrayServer/ArrayServerData/
1. First check if the system already has some swap space available:
sudo swapon --show
If you don’t get back any output, this means your system does not have swap space available currently.
2. Check available disk space on your machine, to identify the best location for your swap space
> df -h
Filesystem Size Used Avail Use% Mounted on udev 16G 0 16G 0% /dev tmpfs. 3.1G 8.5M 3.1G 1% /run /dev/nvme0n1p1 30G 14G 16G 48% / /dev/nvme1n1 886G 735G 112G 87% /opt/ArrayServer/ArrayServerData
Notice that the root device /dev/nvm0n1p1 only has 16G space. If you make a large swap space, you could use up the entire drive.
Notice that /dev/nvme1n1, the second disk, has plenty of space. Make swap space here.
3. Based on the total memory of your machine create a swap file within the filesystem:
sudo fallocate -l 6G /opt/ArrayServer/ArrayServerData/swapfile
Please modify it to a value based on your system (one can use the recommendations from here. Eg. For 32GB RAM should start with a 6GB swapfile. )
If you get an error ***fallocate failed: Operation not supported*** you should instead use the command below (where count) would be the size in kilobytes, so the example below would be 8 GB
dd if=/dev/zero of=/swapfile bs=1024 count=8388608
4. Make the file only accessible to root by typing:
sudo chmod 600 /opt/ArrayServer/ArrayServerData/swapfile
5. Mark the file as swap space:
sudo mkswap /opt/ArrayServer/ArrayServerData/swapfile
6. Enable the swap file:
sudo swapon /opt/ArrayServer/ArrayServerData/swapfile
7. Make the Swap file Permanent (in case a system reboot the server will not retain the swap settings automatically, so we need the following):
Backup fstab file first:
sudo cp /etc/fstab /etc/fstab.bak
Add the swap information to the end of /etc/fstab file:
echo '/opt/ArrayServer/ArrayServerData/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
Q. Oh no! I'm getting a Disk Space Full error!
A. Not to worry. You probably specified your swap file on a boot drive with little space, or some other location that was too small for your swap file. Just delete the file and choose a different location.