notebooks/ElasticBLAST/run_elastic_blast.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8c3f3bb2",
   "metadata": {},
   "source": [
    "# Run ElasticBLAST using AWS Batch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af8ad165",
   "metadata": {},
   "source": [
    "## Overview\n",
    "This notebook helps you to run Blast in a scalable manner using AWS Batch. The script will spin up and later tear down your cluster to execute the Blast jobs. This notebook is based on the [this tutorial](https://blast.ncbi.nlm.nih.gov/doc/elastic-blast/quickstart-aws.html). Make sure you select a kernel with Python 3.7 for the Elastic BLAST install. One good option is `conda_mxnet_latest_p37`. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1855fc00",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "You need to make sure you have permissions use to use Cloud Formation, Batch, and SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13e66dc9",
   "metadata": {},
   "source": [
    "## Learning Objectives\n",
    "+ Learn to use Batch to scale compute jobs.\n",
    "+ Learn how to use BLAST in the cloud."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3c61097",
   "metadata": {},
   "source": [
    "## Get Started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e31b2a",
   "metadata": {},
   "source": [
    "### Install packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d96bb988",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip3 install elastic-blast"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "684e79f6",
   "metadata": {},
   "source": [
    "Test your install, it should print out a version and full help menu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2aa11ccc",
   "metadata": {},
   "outputs": [],
   "source": [
    "! elastic-blast --version\n",
    "! elastic-blast --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58b59cb0",
   "metadata": {},
   "source": [
    "### Create a bucket for this tutorial if one does not yet exist, make sure to pick a unique name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "319ff226",
   "metadata": {},
   "outputs": [],
   "source": [
    "! aws s3 mb s3://elasticblast-sagemaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "449d7511",
   "metadata": {},
   "source": [
    "### Create a config file that defines the job parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b578c1ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "! touch BDQA.ini"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1b0a866",
   "metadata": {},
   "source": [
    "Open the config file and add the following:\n",
    "```\n",
    "[cloud-provider]\n",
    "aws-region = us-east-1\n",
    "aws-vpc = vpc-0eaafe0236e351a36\n",
    "aws-subnet = subnet-043d7614ae5dc30c9\n",
    "aws-key-pair = cloud-lab-testing\n",
    "\n",
    "[cluster]\n",
    "num-nodes = 3\n",
    "labels = owner=ec2-user\n",
    "\n",
    "[blast]\n",
    "program = blastp\n",
    "db = refseq_protein\n",
    "queries = s3://elasticblast-test/queries/BDQA01.1.fsa_aa\n",
    "results = s3://elasticblast-sagemaker/results/BDQA\n",
    "options = -task blastp-fast -evalue 0.01 -outfmt \"7 std sskingdoms ssciname\"\n",
    "```\n",
    "\n",
    "You can add additional configuration values from [this guide](https://blast.ncbi.nlm.nih.gov/doc/elastic-blast/configuration.html). If you need to run this a few times, make sure you either rename the ouput folder, or delete the results folder from the S3 bucket. If you are using your own data, make sure to modify the database and the S3 queries path."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9f8192",
   "metadata": {},
   "source": [
    "### Submit the job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "398253e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "! elastic-blast submit --cfg BDQA.ini"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a8e7716",
   "metadata": {},
   "source": [
    "### Check results and troubleshoot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a43c5e",
   "metadata": {},
   "source": [
    "+ You can monitor the job initially by going to `CloudFormation` and viewing the events tab of the elastic blast stack. If there is an error, you should be able to pinpoint it in these event logs.\n",
    "+ You can view the progress by going to `AWS Batch`, select the Job queue that begins with `elasticblast`, and then make sure jobs are moving from Runnable to Running to Succeeded. The number of jobs that run together will be the number of nodes you selected in the config file. To run more jobs at once, increase the `cluster` parameter `num-nodes`. \n",
    "+ Finally, to view your outputs, look at the files in your S3 output bucket, something like `aws s3 ls s3://elasticblast-sagemaker/results/BDQA/`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d27edaa5",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "Here we submited a parallel Blast job to an AWS Batch cluster using Cloud Formation to handle provisioning and tear down of resources. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "292947f1-5247-4da5-81bd-7fc8fc420ca4",
   "metadata": {},
   "source": [
    "## Clean Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e677ba64-38e0-49d4-919b-4bb51de83cdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "! elastic-blast delete --cfg BDQA.ini"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}