machine_learning_workflow_abalone.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a machine learning workflow using Step Functions and SageMaker\n",
    "\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Build a machine learning workflow](#Build-a-machine-learning-workflow)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook describes using the AWS Step Functions Data Science SDK to create and manage workflows. The Step Functions SDK is an open source library that allows data scientists to easily create and execute machine learning workflows using AWS Step Functions and Amazon SageMaker. For more information, see the following.\n",
    "* [AWS Step Functions](https://aws.amazon.com/step-functions/)\n",
    "* [AWS Step Functions Developer Guide](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html)\n",
    "* [AWS Step Functions Data Science SDK](https://aws-step-functions-data-science-sdk.readthedocs.io)\n",
    "\n",
    "In this notebook we will use the SDK to create steps, link them together to create a workflow, and execute the workflow in AWS Step Functions. The first tutorial shows how to create an ML pipeline workflow, and the second shows how to run multiple experiments in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install --upgrade stepfunctions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "### Add a policy to your SageMaker role in IAM\n",
    "\n",
    "**If you are running this notebook on an Amazon SageMaker notebook instance**, the IAM role assumed by your notebook instance needs permission to create and run workflows in AWS Step Functions. To provide this permission to the role, do the following.\n",
    "\n",
    "1. Open the Amazon [SageMaker console](https://console.aws.amazon.com/sagemaker/). \n",
    "2. Select **Notebook instances** and choose the name of your notebook instance\n",
    "3. Under **Permissions and encryption** select the role ARN to view the role on the IAM console\n",
    "4. Choose **Attach policies** and search for `AWSStepFunctionsFullAccess`.\n",
    "5. Select the check box next to `AWSStepFunctionsFullAccess` and choose **Attach policy**\n",
    "\n",
    "If you are running this notebook in a local environment, the SDK will use your configured AWS CLI configuration. For more information, see [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html).\n",
    "\n",
    "Next, create an execution role in IAM for Step Functions. \n",
    "\n",
    "### Create an execution role for Step Functions\n",
    "\n",
    "You need an execution role so that you can create and execute workflows in Step Functions.\n",
    "\n",
    "1. Go to the [IAM console](https://console.aws.amazon.com/iam/)\n",
    "2. Select **Roles** and then **Create role**.\n",
    "3. Under **Choose the service that will use this role** select **Step Functions**\n",
    "4. Choose **Next** until you can enter a **Role name**\n",
    "5. Enter a name such as `StepFunctionsWorkflowExecutionRole` and then select **Create role**\n",
    "\n",
    "\n",
    "Attach a policy to the role you created. The following steps attach a policy that provides full access to Step Functions, however as a good practice you should only provide access to the resources you need.  \n",
    "\n",
    "1. Under the **Permissions** tab, click **Add inline policy**\n",
    "2. Enter the following in the **JSON** tab\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"sagemaker:CreateTransformJob\",\n",
    "                \"sagemaker:DescribeTransformJob\",\n",
    "                \"sagemaker:StopTransformJob\",\n",
    "                \"sagemaker:CreateTrainingJob\",\n",
    "                \"sagemaker:DescribeTrainingJob\",\n",
    "                \"sagemaker:StopTrainingJob\",\n",
    "                \"sagemaker:CreateHyperParameterTuningJob\",\n",
    "                \"sagemaker:DescribeHyperParameterTuningJob\",\n",
    "                \"sagemaker:StopHyperParameterTuningJob\",\n",
    "                \"sagemaker:CreateModel\",\n",
    "                \"sagemaker:CreateEndpointConfig\",\n",
    "                \"sagemaker:CreateEndpoint\",\n",
    "                \"sagemaker:DeleteEndpointConfig\",\n",
    "                \"sagemaker:DeleteEndpoint\",\n",
    "                \"sagemaker:UpdateEndpoint\",\n",
    "                \"sagemaker:ListTags\",\n",
    "                \"lambda:InvokeFunction\",\n",
    "                \"sqs:SendMessage\",\n",
    "                \"sns:Publish\",\n",
    "                \"ecs:RunTask\",\n",
    "                \"ecs:StopTask\",\n",
    "                \"ecs:DescribeTasks\",\n",
    "                \"dynamodb:GetItem\",\n",
    "                \"dynamodb:PutItem\",\n",
    "                \"dynamodb:UpdateItem\",\n",
    "                \"dynamodb:DeleteItem\",\n",
    "                \"batch:SubmitJob\",\n",
    "                \"batch:DescribeJobs\",\n",
    "                \"batch:TerminateJob\",\n",
    "                \"glue:StartJobRun\",\n",
    "                \"glue:GetJobRun\",\n",
    "                \"glue:GetJobRuns\",\n",
    "                \"glue:BatchStopJobRun\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        },\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"iam:PassRole\"\n",
    "            ],\n",
    "            \"Resource\": \"*\",\n",
    "            \"Condition\": {\n",
    "                \"StringEquals\": {\n",
    "                    \"iam:PassedToService\": \"sagemaker.amazonaws.com\"\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"events:PutTargets\",\n",
    "                \"events:PutRule\",\n",
    "                \"events:DescribeRule\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTrainingJobsRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTransformJobsRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTuningJobsRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForECSTaskRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForBatchJobsRule\"\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```\n",
    "\n",
    "3. Choose **Review policy** and give the policy a name such as `StepFunctionsWorkflowExecutionPolicy`\n",
    "4. Choose **Create policy**. You will be redirected to the details page for the role.\n",
    "5. Copy the **Role ARN** at the top of the **Summary**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure execution roles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "# SageMaker Execution Role\n",
    "# You can use sagemaker.get_execution_role() if running inside sagemaker's notebook instance\n",
    "sagemaker_execution_role = sagemaker.get_execution_role() #Replace with ARN if not in an AWS SageMaker notebook\n",
    "\n",
    "# paste the StepFunctionsWorkflowExecutionRole ARN from above\n",
    "workflow_execution_role = \"<execution-role-arn>\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the required modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import time\n",
    "import random\n",
    "import uuid\n",
    "import logging\n",
    "import stepfunctions\n",
    "import io\n",
    "import random\n",
    "\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "from stepfunctions import steps\n",
    "from stepfunctions.steps import TrainingStep, ModelStep, TransformStep\n",
    "from stepfunctions.inputs import ExecutionInput\n",
    "from stepfunctions.workflow import Workflow\n",
    "from stepfunctions.template import TrainingPipeline\n",
    "from stepfunctions.template.utils import replace_parameters_with_jsonpath\n",
    "\n",
    "session = sagemaker.Session()\n",
    "stepfunctions.set_stream_logger(level=logging.INFO)\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "bucket = session.default_bucket()\n",
    "prefix = 'sagemaker/DEMO-xgboost-regression'\n",
    "bucket_path = 'https://s3-{}.amazonaws.com/{}'.format(region, bucket)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the dataset\n",
    "\n",
    "The following cell defines utility methods to split a dataset into train, validation, and test datasets. It then defines methods to upload them to an Amazon S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_split(FILE_DATA, FILE_TRAIN, FILE_VALIDATION, FILE_TEST, PERCENT_TRAIN, PERCENT_VALIDATION, PERCENT_TEST):\n",
    "    data = [l for l in open(FILE_DATA, 'r')]\n",
    "    train_file = open(FILE_TRAIN, 'w')\n",
    "    valid_file = open(FILE_VALIDATION, 'w')\n",
    "    tests_file = open(FILE_TEST, 'w')\n",
    "\n",
    "    num_of_data = len(data)\n",
    "    num_train = int((PERCENT_TRAIN/100.0)*num_of_data)\n",
    "    num_valid = int((PERCENT_VALIDATION/100.0)*num_of_data)\n",
    "    num_tests = int((PERCENT_TEST/100.0)*num_of_data)\n",
    "\n",
    "    data_fractions = [num_train, num_valid, num_tests]\n",
    "    split_data = [[],[],[]]\n",
    "\n",
    "    rand_data_ind = 0\n",
    "\n",
    "    for split_ind, fraction in enumerate(data_fractions):\n",
    "        for i in range(fraction):\n",
    "            rand_data_ind = random.randint(0, len(data)-1)\n",
    "            split_data[split_ind].append(data[rand_data_ind])\n",
    "            data.pop(rand_data_ind)\n",
    "\n",
    "    for l in split_data[0]:\n",
    "        train_file.write(l)\n",
    "\n",
    "    for l in split_data[1]:\n",
    "        valid_file.write(l)\n",
    "\n",
    "    for l in split_data[2]:\n",
    "        tests_file.write(l)\n",
    "\n",
    "    train_file.close()\n",
    "    valid_file.close()\n",
    "    tests_file.close()\n",
    "\n",
    "def write_to_s3(fobj, bucket, key):\n",
    "    return boto3.Session(region_name=region).resource('s3').Bucket(bucket).Object(key).upload_fileobj(fobj)\n",
    "\n",
    "def upload_to_s3(bucket, channel, filename):\n",
    "    fobj=open(filename, 'rb')\n",
    "    key = prefix+'/'+channel\n",
    "    url = 's3://{}/{}/{}'.format(bucket, key, filename)\n",
    "    print('Writing to {}'.format(url))\n",
    "    write_to_s3(fobj, bucket, key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook uses the XGBoost algorithm to train and host a regression model. We use the [Abalone data](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html) originally from the [UCI data repository](https://archive.ics.uci.edu/ml/datasets/abalone). More details about the original dataset can be found [here](https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.names).  In the libsvm converted [version](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html), the nominal feature (Male/Female/Infant) has been converted into a real valued feature. Age of abalone is to be predicted from eight physical measurements.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try: #python3\n",
    "    from urllib.request import urlretrieve\n",
    "except: #python2\n",
    "    from urllib import urlretrieve\n",
    "    \n",
    "# Load the dataset\n",
    "FILE_DATA = 'abalone'\n",
    "urlretrieve(\"https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression/abalone\", FILE_DATA)\n",
    "\n",
    "#split the downloaded data into train/test/validation files\n",
    "FILE_TRAIN = 'abalone.train'\n",
    "FILE_VALIDATION = 'abalone.validation'\n",
    "FILE_TEST = 'abalone.test'\n",
    "PERCENT_TRAIN = 70\n",
    "PERCENT_VALIDATION = 15\n",
    "PERCENT_TEST = 15\n",
    "data_split(FILE_DATA, FILE_TRAIN, FILE_VALIDATION, FILE_TEST, PERCENT_TRAIN, PERCENT_VALIDATION, PERCENT_TEST)\n",
    "\n",
    "#upload the files to the S3 bucket\n",
    "upload_to_s3(bucket, 'train', FILE_TRAIN)\n",
    "upload_to_s3(bucket, 'validation', FILE_VALIDATION)\n",
    "upload_to_s3(bucket, 'test', FILE_TEST)\n",
    "\n",
    "train_s3_file = bucket_path + \"/\" + prefix + '/train'\n",
    "validation_s3_file = bucket_path + \"/\" + prefix + '/validation'\n",
    "test_s3_file = bucket_path + \"/\" + prefix + '/test'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the AWS Sagemaker estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb = sagemaker.estimator.Estimator(\n",
    "    get_image_uri(region, 'xgboost'),\n",
    "    sagemaker_execution_role, \n",
    "    train_instance_count = 1, \n",
    "    train_instance_type = 'ml.m4.4xlarge',\n",
    "    train_volume_size = 5,\n",
    "    output_path = bucket_path + \"/\" + prefix + \"/single-xgboost\",\n",
    "    sagemaker_session = session\n",
    ")\n",
    "\n",
    "xgb.set_hyperparameters(\n",
    "    objective = 'reg:linear',\n",
    "    num_round = 50,\n",
    "    max_depth = 5,\n",
    "    eta = 0.2,\n",
    "    gamme = 4,\n",
    "    min_child_weight = 6,\n",
    "    subsample = 0.7,\n",
    "    silent = 0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Build a machine learning workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/e2e_pipeline.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use a workflow to create a machine learning pipeline. The AWS Data Science Workflows SDK provides several AWS SageMaker workflow steps that you can use to construct an ML pipeline. In this tutorial you will use the Train and Transform steps.\n",
    "\n",
    "* [**TrainingStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.TrainingStep) - Starts a Sagemaker training job and outputs the model artifacts to S3.\n",
    "* [**ModelStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.ModelStep) - Creates a model on SageMaker using the model artifacts from S3.\n",
    "* [**TransformStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.TransformStep) - Starts a SageMaker transform job\n",
    "* [**EndpointConfigStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.EndpointConfigStep) - Defines an endpoint configuration on SageMaker.\n",
    "* [**EndpointStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.EndpointStep) - Deploys the trained model to the configured endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the input schema for a workflow execution\n",
    "\n",
    "The [**ExecutionInput**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/placeholders.html#stepfunctions.inputs.ExecutionInput) API defines the options to dynamically pass information to a workflow at runtime.\n",
    "\n",
    "The following cell defines the fields that must be passed to your workflow when starting an execution.\n",
    "\n",
    "While the workflow is usually static after it is defined, you may want to pass values dynamically that are used by steps in your workflow. To help with this, the SDK provides a way to create placeholders when you define your workflow. These placeholders can be dynamically assigned values when you execute your workflow.\n",
    "\n",
    "ExecutionInput values are accessible to each step of your workflow. You have the ability to define a schema for this placeholder collection, as shown in the cell below. When you execute your workflow the SDK will verify if the dynamic input conforms to the schema you defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SageMaker expects unique names for each job, model and endpoint. \n",
    "# If these names are not unique the execution will fail. Pass these\n",
    "# dynamically for each execution using placeholders.\n",
    "execution_input = ExecutionInput(schema={\n",
    "    'JobName': str, \n",
    "    'ModelName': str,\n",
    "    'EndpointName': str\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the training step \n",
    "\n",
    "In the following cell we create the training step and pass the estimator we defined above. See  [TrainingStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.TrainingStep) in the AWS Step Functions Data Science SDK documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_step = steps.TrainingStep(\n",
    "    'Train Step', \n",
    "    estimator=xgb,\n",
    "    data={\n",
    "        'train': sagemaker.s3_input(train_s3_file, content_type='libsvm'),\n",
    "        'validation': sagemaker.s3_input(validation_s3_file, content_type='libsvm')\n",
    "    },\n",
    "    job_name=execution_input['JobName']  \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the model step \n",
    "\n",
    "In the following cell we define a model step that will create a model in SageMaker using the artifacts created during the TrainingStep. See  [ModelStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.ModelStep) in the AWS Step Functions Data Science SDK documentation.\n",
    "\n",
    "The model creation step typically follows the training step. The Step Functions SDK provides the [get_expected_model](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.TrainingStep.get_expected_model) method in the TrainingStep class to provide a reference for the trained model artifacts. Please note that this method is only useful when the ModelStep directly follows the TrainingStep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_step = steps.ModelStep(\n",
    "    'Save model',\n",
    "    model=training_step.get_expected_model(),\n",
    "    model_name=execution_input['ModelName']  \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the transform step\n",
    "\n",
    "In the following cell we create the transform step. See  [TransformStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.TransformStep) in the AWS Step Functions Data Science SDK documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform_step = steps.TransformStep(\n",
    "    'Transform Input Dataset',\n",
    "    transformer=xgb.transformer(\n",
    "        instance_count=1,\n",
    "        instance_type='ml.m5.large'\n",
    "    ),\n",
    "    job_name=execution_input['JobName'],     \n",
    "    model_name=execution_input['ModelName'], \n",
    "    data=test_s3_file,\n",
    "    content_type='text/libsvm'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an endpoint configuration step\n",
    "\n",
    "In the following cell we create an endpoint configuration step. See [EndpointConfigStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.EndpointConfigStep) in the AWS Step Functions Data Science SDK documentation.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_config_step = steps.EndpointConfigStep(\n",
    "    \"Create Endpoint Config\",\n",
    "    endpoint_config_name=execution_input['ModelName'],\n",
    "    model_name=execution_input['ModelName'],\n",
    "    initial_instance_count=1,\n",
    "    instance_type='ml.m5.large'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an endpoint\n",
    "\n",
    "In the following cell we create a step to deploy the trained model to an endpoint in AWS SageMaker. See [EndpointStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.sagemaker.EndpointStep) in the AWS Step Functions Data Science SDK documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_step = steps.EndpointStep(\n",
    "    \"Create Endpoint\",\n",
    "    endpoint_name=execution_input['EndpointName'],\n",
    "    endpoint_config_name=execution_input['ModelName']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chain together steps for your workflow\n",
    "\n",
    "Create your workflow definition by chaining the steps together. See [Chain](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.html#stepfunctions.steps.states.Chain) in the AWS Step Functions Data Science SDK documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow_definition = steps.Chain([\n",
    "    training_step,\n",
    "    model_step,\n",
    "    transform_step,\n",
    "    endpoint_config_step,\n",
    "    endpoint_step\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create your workflow using the workflow definition above, and render the graph with [render_graph](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.html#stepfunctions.workflow.Workflow.render_graph)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow = Workflow(\n",
    "    name='MyTrainTransformDeploy_v1',\n",
    "    definition=workflow_definition,\n",
    "    role=workflow_execution_role,\n",
    "    execution_input=execution_input\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow.render_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the workflow in AWS Step Functions with [create](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.html#stepfunctions.workflow.Workflow.create)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow.create()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the workflow with [execute](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.html#stepfunctions.workflow.Workflow.execute)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "execution = workflow.execute(\n",
    "    inputs={\n",
    "        'JobName': 'regression-{}'.format(uuid.uuid1().hex), # Each Sagemaker Job requires a unique name\n",
    "        'ModelName': 'regression-{}'.format(uuid.uuid1().hex), # Each Model requires a unique name,\n",
    "        'EndpointName': 'regression-{}'.format(uuid.uuid1().hex) # Each Endpoint requires a unique name,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Render workflow progress with the [render_progress](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.html#stepfunctions.workflow.Execution.render_progress).\n",
    "\n",
    "This generates a snapshot of the current state of your workflow as it executes. This is a static image. Run the cell again to check progress. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "execution.render_progress()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use [list_events](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.html#stepfunctions.workflow.Execution.list_events) to list all events in the workflow execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "execution.list_events(html=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use [list_executions](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.html#stepfunctions.workflow.Workflow.list_executions) to list all executions for a specific workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow.list_executions(html=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use [list_workflows](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.html#stepfunctions.workflow.Workflow.list_workflows) to list all workflows in your AWS account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Workflow.list_workflows(html=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}