EnableAWSSpot

From Array Suite Wiki

(Difference between revisions)
Jump to: navigation, search
Line 51: Line 51:
  
 
==== Supported Procs ====
 
==== Supported Procs ====
This is a list of all procs that support <code>/[[EnableAWSSpot]]</code> and <code>/[[SpotBidMaxPrice]]</code> as oscript parameters.
+
Please note that '''all procs''' that can run on AWS Cloud will benefit from spot instances if <code>EnableAWSSpot</code> is set to '''<code>True</code>''' in [[ArrayServer.cfg]].
 +
 
 +
In addition, the following list of procs support <code>/[[EnableAWSSpot]]</code> and <code>/[[SpotBidMaxPrice]]</code> as oscript parameters.
 
<br />
 
<br />
Please note that '''all procs''' will benefit from spot instances if <code>EnableAWSSpot</code> is set to '''<code>True</code>''' in [[ArrayServer.cfg]].
+
 
 
<br />
 
<br />
 
* ProcConvertNgsFiles
 
* ProcConvertNgsFiles

Revision as of 16:18, 28 August 2019


Contents

Definition

Flag that tells the application whether or not Spot Instances should be used.
When Spot Instances are used, you pay the Spot price that's in effect for the time period your instances are running (more details here).

Possible values:

  • False (default - OnDemand instances are used)
  • True (Spot instances are used)

Enabling spot instances in script

If the user wants to enable spot instances for any type of analysis, the user can set /EnableAWSSpot=True in the oscript Options section. This will overwrite the default option specified in ArrayServer.cfg for the duration of these jobs.

Specifying spot instance price in script

Warning.png WARNING: We recommend using default price as the on demand price (as max price) to minimize job termination risk.


Additionally, only in the oscript, in the Options section, when /EnableAWSSpot is set to True, user can also specify a spot price she is willing to pay, with parameter /SpotBidMaxPrice=x.xxxx. This will have an effect only for the duration of these jobs. If spot instances are more expensive at the time of the run, the oscript will fail.

e.g. /EnableAWSSpot=True /SpotBidMaxPrice=0.045

Pricing Examples

We explore two scenarios, starting from the assumption that user runs one spot instance for two hours, and after one hour, the current spot price doubles.

Scenario 1

Description: User sets a valid spot price (i.e. higher than the current spot price).
Parameters /EnableAWSSpot=True /SpotBidMaxPrice=0.045
Your spot price = $0.045
Spot price at start of run = $0.025
OnDemand price = $0.125
Spot price after one hour of run = $0.05
Outcome: You pay $0.025 for the first hour of run, then the instance is terminated by Amazon because your spot price is lower than Spot price after one hour run.

Scenario 2

Description: User does not set any spot price.
Parameters /EnableAWSSpot=True
Your spot price = $0.125 (not specified by user, but Amazon will assume it is OnDemand price)
Spot price at start of run = $0.025
OnDemand price = $0.125
Spot price after one hour of run = $0.05
Outcome: You pay $0.025 for the first hour of run and $0.05 for the second hour of run.


One has to consider that no matter what spot price you specify (or don't specify at all), you always pay the current spot price at the time of your run. So, it does not matter that in Scenario 1 user specified a smaller spot price than the one in Scenario 2, because in both situations, for the first hour of run, the charges were the same. More to this, in Scenario 1, user is at risk of having its instances terminated by Amazon and losing all work, if current spot price rises above user's initial spot price. On the other hand, if you do not specify any spot price, you are going to pay at most the OnDemand price (more details here).

Supported Procs

Please note that all procs that can run on AWS Cloud will benefit from spot instances if EnableAWSSpot is set to True in ArrayServer.cfg.

In addition, the following list of procs support /EnableAWSSpot and /SpotBidMaxPrice as oscript parameters.


  • ProcConvertNgsFiles
  • ProcCountErcc
  • ProcDemultiplexNgsFiles
  • ProcExportNgsData
  • ProcFilterNgsFiles
  • ProcMapLongRnaSeqReads
  • ProcMergeNgsFiles
  • ProcPreviewDnaSeqReads
  • ProcPreviewLongDnaSeqReads
  • ProcPreviewLongRnaSeqReads
  • ProcPreviewMirnaSeqReads
  • ProcPreviewRnaSeqReads
  • ProcPreviewRnaSeqReadsToTranscriptome
  • ProcReportPairedEndFusionGenes
  • ProcRestoreUnoptimizedBamFiles
  • ProcSampleNgsFiles
  • ProcSCLandTools
  • ProcSearchNgsAdapters
  • ProcSearchNgsTags
  • ProcSentieonDnaSeqPipeline
  • ProcSentieonTNSeqPipeline
  • ProcSplitNgsFileByTag
  • ProcSummarizeFlagStatistics

Common Errors

Price too low: if you see an instance creation error, you will be notified if attempt was unsuccessful because your bid was too low.

[00:00:40] Error occurred from OJobProcess.Run. Error message=Instance creation is not successful.
Error=Your Spot request price of 0.3 is lower than the minimum required Spot request fulfillment price of 0.563.

Price is zero or negative: if you set the price to /SpotBidMaxPrice=0.0000 or /SpotBidMaxPrice=-0.5000, Amazon will set the default, OnDemand price for you.

Potentially useful page for Spot Instance Pricing https://aws.amazon.com/ec2/spot/pricing/

Related Articles