Sagemaker_RL_Lab_Summit_2019_One_Click_old.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0Zv-79trOkkx"
   },
   "source": [
    "# Sagemaker Sydney Summit 2019 Reinforcement Learning Lab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Authors: Paul Conyngham & William Xu | StarAi Project - Machine Learning Engineers - | Email : contact@starai.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](https://cdn-images-1.medium.com/max/1200/1*XyIpmXXAjbXerDzmGQL1yA.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hi! and welcome to the sage maker Reinforcement Learning lab for AWS Summit 2019. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This workshop gives you an introduction to state of the art Reinforcement Learning & shows you how, using AWS, you can parallelise the training of your reinforcement learning algorithms to get insanely fast turn around times & results for your reinforcement learning experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reinforcement Learning has been directly responsible for some of the biggest breakthoughs in the field of modern machine learning. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the events you may have heard about in the media include [AlphaGo](https://deepmind.com/research/alphago/), [OpenAI Five](https://openai.com/blog/openai-five/) &[ Deepmind's AlphaStar](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/) - all powered by Reinforcement Learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outside of games however, we are beginning to see more use cases of Reinforcement Learning in the wild. \n",
    "\n",
    "Including [web stores product recommendations](https://arxiv.org/pdf/1805.02343.pdf),  [  news recommendation](https://towardsdatascience.com/deep-reinforcement-learning-for-news-recommendation-part-1-architecture-5741b1a6ed56), [in the finance industry](https://www.forbes.com/sites/quora/2018/07/25/what-are-the-latest-works-on-reinforcement-learning-in-the-financial-field/#3c35b15b44df) , [Automated Architecture search for Machine Learning using RL](https://ai.googleblog.com/2017/05/using-machine-learning-to-explore.html) ,[End to End  Machine Learning for Self Driving Cars](https://wayve.ai/), and last and most excitingly the field of Robotics where [Deep Reinforcement Learning is exploding!](https://www.therobotreport.com/reinforcement-learning-industrial-robotics/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But first, an intro to Jupyter notebooks! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gbqo0hk7Ornl"
   },
   "source": [
    "\n",
    "**For those new to Jupyter Notebooks** - the interface you are using right now - there are two types of \"cell blocks\" **text** (like the one you are reading) and **code blocks**.\n",
    "\n",
    "To run code Cell blocks :\n",
    "\n",
    "1.  First select the code cell by simply left clicking on it below, when you select it - it should get a green border as follows:\n",
    "\n",
    "![alt text](https://i.ibb.co/rt9NdXL/Screen-Shot-2019-03-25-at-2-02-08-pm.png)\n",
    "\n",
    "2.   Once the cell is selected, click the \"run\" button at the top of the notebook, which looks like this:\n",
    "\n",
    "![alt text](https://i.ibb.co/fCwfvBk/Screen-Shot-2019-03-25-at-2-05-33-pm.png)\n",
    "\n",
    "Hitting \"shift-enter\"together is also a shortcut for the Run command/button above\n",
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to get started with training as soon as possible, select the code cell directly below where is says \"run the code cell below\" and run it. If you need help follow step 1 & 2 above to do this\n",
    "\n",
    "\n",
    "This will start our reinforcement learning model's training process. We will spend the rest of this lab explaining what is going on here line by line - with an amazing result to follow!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**run the code cell below:**\n",
    "\n",
    "Once the cell block has run, continue scrolling & reading to continue the lesson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ONE CLICK REINFORCEMENT LEARNING!\n",
    "\n",
    "# RUN ME BY FIRST SELECTING THIS BLOCK, AND CLICKING \"RUN\" ABOVE!\n",
    "\"\"\"\n",
    "### Import relevant libraries\n",
    "\n",
    "First we imported the relevant Python libraries, you can re do this\n",
    "\"\"\"\n",
    "import sagemaker\n",
    "import boto3\n",
    "import sys\n",
    "import os\n",
    "import glob\n",
    "import re\n",
    "import subprocess\n",
    "from IPython.display import HTML, Markdown\n",
    "import time\n",
    "from time import gmtime, strftime\n",
    "sys.path.append(\"common\")\n",
    "from misc import get_execution_role, wait_for_s3_object\n",
    "from docker_utils import build_and_push_docker_image\n",
    "from sagemaker.rl import RLEstimator, RLToolkit, RLFramework\n",
    "from markdown_helper import generate_help_for_s3_endpoint_permissions, create_s3_endpoint_manually\n",
    "\n",
    "gym_problem = 'breakout'\n",
    "\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "### Setup S3 bucket\n",
    "\n",
    "Set up the linkage and authentication to the S3 bucket. \n",
    "S3 Buckets are a form of data storage available on AWS. \n",
    "Our bucket will be used for checkpointing and the metadata. \n",
    "\"\"\"\n",
    "sage_session = sagemaker.session.Session()\n",
    "s3_bucket = sage_session.default_bucket()  \n",
    "s3_output_path = 's3://{}/'.format(s3_bucket)\n",
    "print(\"S3 bucket path: {}\".format(s3_output_path))\n",
    "\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "### Define Variables \n",
    "\n",
    "We define variables such as the job prefix for the training jobs \n",
    "and the image path for the container (only when this is BYOC).\n",
    "\"\"\"\n",
    "job_name_prefix = 'rl-gym-summit'\n",
    "aws_region = boto3.Session().region_name\n",
    "\n",
    "instance_type = \"ml.p3.8xlarge\"\n",
    "train_instance_count = 1\n",
    "\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "### Create an IAM role\n",
    "\n",
    "Either get the execution role when running from a SageMaker notebook instance \n",
    "`role = sagemaker.get_execution_role()` or, when running from local notebook instance, \n",
    "use utils method `role = get_execution_role()` to create an execution role.\n",
    "\"\"\"\n",
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except:\n",
    "    role = get_execution_role()\n",
    "\n",
    "print(\"Using IAM role arn: {}\".format(role))\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "### Build docker container for SageMaker Training job.\n",
    "\n",
    "The commented code below will build a container, \n",
    "however it takes some time so we've prebuilt the container for this workshop\n",
    "\n",
    "    cpu_or_gpu = 'gpu' if instance_type.startswith('ml.p') else 'cpu'\n",
    "    repository_short_name = \"sagemaker-gym-ray-%s\" % cpu_or_gpu\n",
    "    docker_build_args = {\n",
    "        'CPU_OR_GPU': cpu_or_gpu, \n",
    "        'AWS_REGION': boto3.Session().region_name,\n",
    "    }\n",
    "    custom_image_name = build_and_push_docker_image(repository_short_name, build_args=docker_build_args)\n",
    "    print(\"Using ECR image %s\" % custom_image_name)    \n",
    "    print(\"the repo name is \", repository_short_name)\n",
    "\"\"\"\n",
    "custom_image_name = \"413953306596.dkr.ecr.us-east-1.amazonaws.com/sagemaker-gym-ray-gpu\"\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "### Start a training job\n",
    "This will run a training job on a a separate VM instance (ml.p3.8xlarge). \n",
    "You can view the status of this training job from the Amazon Sagemaker menu\n",
    "\n",
    "Amazon Sagemaker --> Training --> Training Jobs\n",
    "\"\"\"\n",
    "metric_definitions = RLEstimator.default_metric_definitions(RLToolkit.RAY)\n",
    "    \n",
    "estimator = RLEstimator(entry_point=\"train-%s.py\" % gym_problem,\n",
    "                        source_dir='src',\n",
    "                        dependencies=[\"common/sagemaker_rl\"],\n",
    "                        image_name=custom_image_name,\n",
    "                        role=role,\n",
    "                        train_instance_type=instance_type,\n",
    "                        train_instance_count=train_instance_count,\n",
    "                        output_path=s3_output_path,\n",
    "                        base_job_name=job_name_prefix,\n",
    "                        metric_definitions=metric_definitions,\n",
    "                        hyperparameters={\n",
    "                            # ml.p3.8xlarge has 32 cores. We have to leave 1 core for ray scheduler.\n",
    "                            \"rl.training.config.num_workers\": 32 * train_instance_count - 1,\n",
    "                            \"rl.training.config.num_gpus\": 2\n",
    "                        }\n",
    "                    )\n",
    "\n",
    "estimator.fit(wait=False)\n",
    "job_name = estimator.latest_training_job.job_name\n",
    "print(\"Training job: %s\" % job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Training Job\n",
    "The above cell will run a training job on a a separate VM instance (ml.p3.8xlarge). You can view the status of this training job from the Amazon Sagemaker menu. It will take a few minutes for the instance to be created before training starts. \n",
    "\n",
    "If you like to see the above training job please follow this link\n",
    "\n",
    "<a href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs\" target=\"_blank\">https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explanation!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today's current enthusiam around RL started only 4 years ago, with the release of [Deepmind DQN algorithm in the scientific journal nature in 2015.](https://www.nature.com/articles/nature14236)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For those who may not be aware, in the DQN paper, Deepmind took several classic atari 2600 games and learnt to play them at human level with just raw pixels values from the screen. \n",
    "\n",
    "When learning to play games, you & I as humans learnt to play this exact same way by just observing the raw images that are being produced by the screen!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But what really set DQN apart from previous work, was that in some instances, the DQN algorithm ***went beyond human level!***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of you may have seen this before, but in what has now become a famous example of Ai research work, let us watch DQN learning to play the Atari game of breakout at a super human level!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBAgIBggICAgGBgYHBwcGBwYHCAgGBgcGBwYHCAcHCAYHChALBwgOCQcGDRUNDhERExMTBwsWGBYSGBASExIBBQUFBwYHDwkIDxUNEBIXFRYSEhUVFRUSEh4TFRUVFhUVFhUSFRUVFRUSFRUVFRIVFRUWEhUVFRIVFRUSFRUVFf/AABEIAWgB4AMBIgACEQEDEQH/xAAcAAEBAAIDAQEAAAAAAAAAAAAACAYHAQIFAwT/xABTEAABAwECBgwKBwQIBQUAAAAAAQIDBAURBwgSNrO0BhchMVFUdHWRktHSExQWGDQ1VXFzkxUiMkFSU2FWgZSiIyQlJjNiobFCQ3Ky8ESCo6TT/8QAGwEBAAIDAQEAAAAAAAAAAAAAAAEEAwUGAgf/xAA1EQEAAQMBBgMHAwIHAAAAAAAAAQIDBBEFFBUhUaESMWETFiJBUtHhcZGigbEGMkJiksHw/9oADAMBAAIRAxEAPwCNZpHPe573Oe97le97lVz3Pct7nOcu65VVVW/9ToAAAAAAAAAAAAAAAAAAAAAAAAAAOzI3LvI5UTfuRVu6DqUNipwsdZ1oq5rHXVcKfWajv+QvCBPTkVFuVFReBdxTg2ZjKRtbsmejUa1PE6XcaiNS/Jf9yGswPr4vJ+CTqr2HHgH/AIH9Vewtmrb4Khklip455o6Z0scOQ2+WRkOU2PcS/dVET95hOwjZdatbaMdPV2D9HUrmyOfVOa9UZksVWpc6NE3VuT94Etvjc1L1a5qb16oqJf8AvOGRudvNc67gRV/2KOxpomtsWkyWsb/X0+y1G/8Ap5eA/JiqxMdZ1oZTWOuq4rspqO/5H6gT94B/4H9VeweBf+B/VUo/CjhRdY1qLRMs6jqWJBFP4V7vBuvkyr25KMVP+H/U2LsOrGV9mUdY6CGJ1XTxVCxNa1zWLIl+SjlTdQCJwfptX0qf40ukcfmAAADuyJypejXuThRFVOlDt4vJ+CTqr2G3cFWy61KKx2Q0lhOtSnSaZyVaI9b3OciuZ9WNU3DKlwiW7+yz+rL/APkBO6wPTdVj0ThVqonTcdC0tl8LFsSucscbXLZ1U5UyGorXeKvW7e30UiwAAAAAA7rE66/Jddv33LdcdC0dkVLH9AVa+DiRfoudUVGNRUXxN27fdvkXAAAAAAAAAAAAAAAAAAAAOWqqKioqoqLeipuKipvKi/cpwAAAAAAAAAAAAAAAAAAPQ2N2W6utCmo2ObG+rnipmyPvVjXSvRqOcjd27dNseb5X8fs/qTd0DS4N0eb5X8fs/qTd0eb5X8fs/qTd0DS4N0eb5X8fs/qTd0eb5X8fs/qTd0DS4N0eb5X8fs/qTd0eb5X8fs/qTd0DS5RWKd6utHlcOgUx3zfK/j9n9SbunsWBgkt+gY9lFbUFGyVUdI2HwzWuciXIqpk79wGG4zGc7+R0v/a81ib1tnAlbFbOs9VatJVTq1GLLKk7n5Lfspfk7yXr0n4/N8r+P2f1J+6Bvms8OlC9aZEWr8Wd4ujrslZ/Bf0V9+5dlZJhOwqo2WOtCNLUjoG2fc/wqxeDSVFyFyMnJcq/auPEbg+2VIiImyNEREuRL595P/YPIDZX+0adM3cA7Y1PqWk5emryn5MVH1daPK4dAp8LdwT7IK6JsVZbcFXE13hGxy+Gc1r7lTKRMnfuVek62Dgjt+ga9tFbUFGyVUdI2Hw7Ue5EuRVTJ37gOuGzBva1p22tVRwRy0600EWU6Vka5bMrKTJct/3obb2A2bLR2NQUs6I2opqSGGVqKjkSRjURyI5N9DXPkFss/aP+afujyC2WftH/ADT90CdrV9Kn+PLpHH5jdUmL/aDnK5a+gVzlVyqrJr1VVvVfs8J183yv4/Z/Um7oGlwbo83yv4/Z/Um7o83yv4/Z/Um7oGwMWpf7rxbt39bqv+9p02T1OzNLQqEoI6B1n+FXxZZUi8IsNyXZV7r9+/fMesXBXsiooPA0tuxUsCOc9IovDNYjnfaW7J31P2+QOyv9o06Zu6B+LZDPs3WgqkqIrNSlWmnSdWJDl+A8E7wuT9b7WTlE+KUZU4O9lEsb45NkLZIpGOjkY5Zla5j2q1zVTJ3lRVMb832v4/Z/Um7ANLA3R5vlfx+z+pN3R5vlfx+z+pN3QNLg3R5vlfx+z+pN3R5vlfx+z+pN3QN3bIvUFXzXPqbiKSjJcHGyh8LoXbIGOhfGsLo1WbJWNzclWr9Xeu3DG/N8r+P2f1Ju6BpcG6PN8r+P2f1Ju6PN8r+P2f1Ju6BpcG6PN8r+P2f1Ju6au2YWG+zbTqKGSRk0lK9sbpY0cjHKsbX3ojt276137gPIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABk2CvOSyucaTTtLNIywV5yWVzjSadpZoAAAAAAAAAAAAAAAAAAAAAAAAAAAFFwAC4IAAAAAAAAAAAAAj/AA3512n8dmrxFgEf4b867T+OzV4gMMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABk2CvOSyucaTTtLNIywV5yWVzjSadpZoHm7ILfo7PjZJW1ENHFI/wAGx8qqjXSZKuyUuRd25FX9x4u2VYPtSi6zu6YRjXep6HnBdVlJvAsfbKsH2pRdZ3dG2VYPtSi6zu6RwALH2yrB9qUXWd3RtlWD7Uous7ukcHIFjbZVg+1KLrO7o2yrB9qUXWd3SSK2xauCmiqZqeeKlnu8DUPYrYpcpquTJeu4u4ir+488Cx9sqwfalF1nd0bZVg+1KLrO7pHAAsfbKsH2pRdZ3dG2VYPtSi6z+6RwALG2yrB9qUXWf3TnbKsH2pRdZ/dI4AFj7ZVg+1KLrP7o2yrB9qUXWf3SOABY+2VYPtSi6z+6NsqwfalF1n90jgAWPtlWD7Uous/ujbKsH2pRdZ/dI4AFj7ZVg+1KLrP7o2yrB9qUXWf3SOABY+2VYPtSi6z+6NsqwfalF1n90jgAWPtlWD7Uous/ujbKsH2pRdZ/dI4AFj7ZVg+1KLrP7o2yrB9qUXWf3SOABY+2VYPtSi6z+6NsqwfalF1n90jgAWPtlWD7Uous/unsbHtkNDaDZHUVTDWNiVGyLEqqjHOS9EXKRN9L+giAoTFO9DtLlFPongbvI/w3512n8dmrxFgEf4b867T+OzV4gMMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkuCvOSyucaTTtLOIwwW5x2VzjSadpaDgNN413qeh5wXVZSbykca71PQ84LqspNwAAADlDgAbSwg7LKGq2IWPZ8EqvrqLwHjMSsciMWOkdG7+kVMl31lTeX7zVpyqnAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKExTvQ7S5RT6J5PZQmKd6JaXx4NG4Dd5H+G/Ou0/js1eIsAj/AA3512n8dmrxAYYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADJMF2cdlc40mnaWg4i/BdnHZXONJp2loOA03jXL/Y9Dy9dVlJvKQxrvU9Dy9dVlJvAAAAAAAAAAHNwHAAAAAAAAAAAAHNwHAAAA5uOAABzcBwAAAObjgAUJineiWl8eDRuJ7KExTvRLS+PBo3AbwQj7DfnXafKGavEWChH2G/Ou0+UM1eIDDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZJguzjsrnGk07S0HEX4Ls47K5xpNO0tBwGm8a71PQ8vXVZScEKPxrvU9Dy9dVlJvA3zgi2O0FRYUMs9HSTzOlnRZJImveqJJcl7lS8y3yPsr2fQfJZ2GmNhmEyWzaBlI2khnax8j0kdK9jl8I6+7JRqntsw0zqqJ4hT7qon+NJw/8ASULlm5NUzH93Z420tnU2Kaa4jxaRE/D89P0aurGIk0iIiIiSSIiJvIiPVEQ+aNN/LgNopFV611c1ZFV6ojIbkV63qm6n6nC4CKLj9d1IewcSx+vaWg4Rla8o7w0FkjJN+bRFHx+t+XD2HO0RRcfrupD2EcTx+vaU8Hy/p7w0FkjJN/bRFFx+u6kPYcbRFFx+u6kPYOJ43XtJwfL+nvDS+xKnZJalGx7WyRvqoWPY5L2uasjUVFT70KMXYhZfEKH5LOw8WyMCtJTVUNQ2trHuglZM1rmRZLlY5HIiqiX3bhsb6OT8Sr/oVsjOtV1R4Z/u6PYuHu9uqm/TGszy8p+TSWHSwqOls2nfTU1PTyOqkYr4mIxys8DIuTen3Xohpo3/AIyNIjLIplRV9NTQSGgbjYYlfjtRLndtxTGZV4fLk7sbuHbJNrYMsFdNatkx1ktXUwSPllj8HGyNzESN9yLe7dvMm2iKPj9b8uHsMde0LFFU0zPOPSWK3srJuUxVTHKfLnDQWQEYb92h6Lj9b8uHsG0PRcfrvlw9h54nj9e0vfBsv6e8NBZJxcb+2h6Lj9b8uHsG0PRcfrvlw9g4nj9e0nBsv6e8fdoJGnOSb82h6Lj9d8uHsG0PRcfrflw9g4nj9e0nBsv6e8fdoG45Rpv3aHouP13y4ewbQ9Fx+u+XD2DieP17ScGy/p7x92g8k63G/toei4/W/Lh7BtD0XH675cPYOJ4/XtJwbL+nvH3aCRpzkm/Noei4/XfLh7BtEUXH635cPYOJ4/XtJwbL+nvH3aCuCNN+7Q9Fx+u+XD2DaHouP13y4ewcTx+vaTg2X9PePu0Hkm/8VD0W0vjwaJx02iKLj9b8uHsM9wWbB4bFjqGQzzVKVD2SOWVrWq1WNVtyZG+m6ZLWbZu1eGidZYb+zsixR4640j9YZqhH2G/Ou0+UM1eIsFCPsN+ddp8oZq8RbUWGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyTBdnHZXONJp2loOIvwXZx2VzjSadpaDgNN413qeh5euqyk3lIY13qeh5euqyk3gDsxbnIvAqL/qdTlAmJ0lUcWFqwka1PG33o1E9Hn4E/yHp7HcINlWhVNpqWofLUPRzmsWGViKjG3u+s9qIm4Sb4vJ+CTqu7DP8AML27JqdVa9qeCqEvVqon+Cv3qaXI2fYt2qqomeUdW/sbav1XIpmI0mVOA5FxzftG6jKq+bgHOSoyVI9oneanAO13vF3vJ9obzUwDDbsYq7Us6CGjYySVlUkzke9saIxIntvynffe5DUG05bvF4P4iLtKeu94uL9na1yxR4aYhrMnDt5Fya69dWIYI7CqLOsSKlqmtjqGSzvc1rmvTJe+9v1mrdvIZadrhcU7mRNyqap+a/bu+zpimPk6g7Xe8Xe8x+0e95qdQdrveLvePaG81OoO13vF3vHtDeanUHa73i73k+M3mp1B2u94u948ZvMuoO1wuI9ojeqnU/DbtqwUNJJVVLljp4Uasj0a56ojnI1PqtRVXdVD92W3hTpQwrDe5vkxXXK37MO5el/pEZYx6PaXaaZ8pljvZtVFE1RprD47bdhcbk/h5+4ZXsF2VUVqMmdRSumbA5rJFWN8WS5yXpuSIl+4hGWUpQeKb6JaXKINE86jH2dasV+KnVzuTtW7kW/BVEN3IR9hvzrtPlDNXiLBQj7DfnXafKGavEX2sYYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADJMF2cdlc40mnaWg4i/BdnHZXONJp2loOA03jXep6Hl66rKTeUhjXep6Hl66rKTeAOzN9Pen+51O0SXuT3p/uBeNPTx+DZ/RxfYb/wADfwp+h9WQMRb0ZG1eFGtRelEPMpq1/g2bqfYb9yfhQ+njr+FOhD5zViXtfNX4vjR8p/Z6WSnAnQLjzfHX8KdCDx1/C3oQ87ndTxqz6vSuFx5vjr+FvQnYPHX8LehOwbndONWfV6VycCC5OBDzfHX8LehOweOv4W9Cdg3O6cas+r0rk4EFycCdB5vjr+FvQnYPHX8LehBud041Z9XpXJwJ0C5OBOg83x1/C3oQeOP4W9CExhXesHGbPq9K5OBBcnAh5vjj+FvQg8cfwt6EG5XesHGrPq9K5P8AxBcnAh53jj+FOhDjxx/C3oQbld6wcas+r0rk4EFycCHneOv4W9CGhNneF+2qO2KylgkpGwQTrHGj6Zj3o3IatyuVd3fLNjZV+/ypmOSxY2lbvTpTMqMyf0QXJwJ0ErbeOyD82h/hWdo28tkH5tF/Cs7Sx7vZXWn95+y17f1VTcnAnQLk4E6DSWBbCTalq19RDVvp3RxU6SsSOBsS5fhEbuqm/uXm2/HpP8vQU72zL9mvwzp+6ne2pRZq8NWv9HpXJwILk4EPN8ek/wAvQPHX/wCXqoY4wr0Sw8atf7kW27XTJXVKJLMiJUzol0j0S7wz/uRT8ElXK5qtdJK5q77XPe5q7t+6ircpTVRgisSSR8joJ1fI90jl8ZmS9z3KrluR25uqpi+E3BpZNDYdVVU8MzKiFrFjc6eV6Iqytat7XLcu4qnbWs2zXMUxE6/oxUbSt3KopjXn/wC6tDlB4py/1O0uUQaJ5PhQeKb6HaXKINE82C43ehH2G/Ou0+UM1eIsFCPsN+ddp8oZq8QGGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyTBdnHZXONJp2loOIvwXZx2VzjSadpaDgNOY13qeh5wXVZSbikMa5f7HoeXrqspN4A7Rrc5F4FRehTqAN/x4dKBGonidobiIm/D9yf9Z6uxrC/R11U2mjpK1kj0cqOesKMRGpet9z1UmsyjBfUxxWvE+V7ImIyVFe9Ua1FVi3bqnjC2Zi3ciii5HwzMa89FGvAs6coU15Rs/Kl6zO058pY/ypelnaYP5QUXGqX5rO0fT9Fxql+a3tO391dh9P5/lVnCo+UM48pY/wAqXpZ2jylj/Kl6Wdpg/wBP0XGqX5re0fT9Fxql+a3tJ91dh9P5/lG5U9GceUsf5UvSztHlLH+VL0s7TB/p+i41S/Nb2j6fouNUvzW9o91dh9P5/k3Knozjylj/ACpelnaPKWP8qXpZ2mD/AE/Rcapfmt7R9P0XGqX5re0j3V2H0/n+Tcqej9uyfC5R0FV4vJS1kj0Y2TKY6FG3PvuT6zr79w8vb3s/iVodaDvmqMK9VHNayvieyVngIm5bFRzb0yr0vT3oYkcRl7Mxbd+qmiOUTOnOVqjAs+HnHdQm3tZ/ErQ60HfPR2OYYKOuq2U8dJWse9HKjnuhyURqXr9l15NRk+DKpjhteJ8j2RMRsiK96o1qXsW7dUjC2Vi3ciimuOUzGvOSvAs6co7qd8pI/wAmXpZ2mObL8KNLZr4mS0tXIszXvasaxXIjFRFvynpwoeV5QUXGqX5jO01phnr4Z5aRYZY5kbHMjljcj8m9zLr7t7eU6Xa/+HNk2Maa7MfFrH+rX/tht4NqZ5xLYe3tQcSr+tB3zS2zS2WV9q1dXG18cdTMsrWPuy2orWpcuTufceIDl7ONbszrRGi9axrdqdaYdsr9RedTlDOzto4u1e2C0qpzkc/KpUaiNuv/AMVF+83l5Sx/lyfy9pOuB2thgrJ3TSRwtWDJR0jkYirlpuIqm0PKCh43S/NZ2nTbN2FsvMsRdyf8066/Fp5emrXZOPTXXrLOvKWP8uT+XtHlLH+XJ/L2mC+UFDxul+aztOfKCh43S/NZ2l2f8LbE0ny/5/lg3OnpL5T4dKBj3MWjtBVY5zVVFhuvaqp+P9DHsIOFyjtKyaijipa2KSdrUa+RYlYmTI127kuVfuNQ2gqLPKqKiossioqbyor1uU+BwVOFZoq1iPJdowrNExVEc4clB4p3olpfHg0bieyhMU70S0vjwaNxaW28EI+w3512nyhmrxFgpvkfYb867T5Q3V4gMMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkmC7OOyucaTTtLQcRfguzjsrnGk07S0HAabxrvU9Dy9dVlJvKQxrvU9Dy9dVlJvAAAAdmqdQB3yjjKOoJ1kdsoZR1A1kdsoZR1A1kdsoZR1A1kcqpwAQByhwAO1/uOFU4BOoAAgDlDgAcop2v9x0AHe8XnQAcqcAAChMU70S0vjwaNxPZQmKd6JaXx4NG4DeCb6EfYb867T5Q3V4iwU30I+w3512nyhurxAYYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADJMF2cdlc40mnaWg4i/BdnHZXONJp2loOA03jXep6Hl66rKTeUhjXep6Hl66rKTeAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAChMU70S0vjwaNxPZQmKd6JaXx4NG4DeCb6EfYb867T5Q3V4iwU30I+w3512nyhurxAYYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADJMF2cdlc40mnaWg4i/BdnHZXONJp2loOA03jXep6Hl66rKTeUhjXep6Hl66rKTeAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAChMU70S0vjwaNxPZQmKd6JaXx4NG4DeCb6EfYb867T5Q3V4iwU30I+w3512nyhurxAYYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADJMF2cdlc40mnaWg4i/BdnHZXONJp2loOA03jXep6Hl66rKTeUjjXep6HnBdVlJuAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQmKd6JaXx4NG4nsoTFO9EtL48GjcBvBN8j/DfnXafKG6vEWA3fJAw3512pyhurxAYUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADJMF2cdlc40mnaWg4i/BdnHZXONJp2loOA03jXL/AGPQ8vXVZSbykMa71PQ8vXVZSbwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAUJineiWl8eDRuJ7KExTvRLS+PBo3AbwTfI/w3r/AHrtTlDdXiLATfQj7DfnXafKG6vEBhgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMkwXZx2VzjSadpaDiL8F2cdlc40mnaWg4DTeNd6noeXrqspN5SGNd6noeXrqspN4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKExTvRLS+PBo3E9lCYp3olpfHg0bgN4JvoR9hvzrtPlDdXiLBTfQj7DfnXafKG6vEBhgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMkwXZx2VzjSadpaDiL8F2cdlc40mnaWg4DTeNd6noeXrqspN5SGNd6noeXrqspN4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKExTvRLS+PBo3E9lCYp3olpfHg0bgN4JvoR9hvzrtPlDdXiLBTfQj7DfnXafKG6vEBhgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMkwXZx2VzjSadpaDiL8F2cdlc40mnaWg4DTeNd6noeXrqspN5SGNd6noeXrqspN4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKExTvRLS+PBo3E9lCYp3olpfHg0bgN4JvoR9hvzrtPlDdXiLBTfI+w3512nyhurxAYYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADJMF2cdlc40mnaWg4i/BdnHZXONJp2loOA05jXep6HnBdVlJuKQxrl/seh5euqyk3gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAoTFO9EtL48GjcT2UJineiWl8eDRuA3g3fJAw3512pyhurxFfpvkf4b1/vXanKG6vEBhYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMkwXZx2VzjSadpaDiL8F2cdlc40mnaWg4DTeNd6noeXrq0pN5SGNd6ooeXrq0pN4AAAAAAAAAAAAcod8kD5g+vgxkIB8gfXIS79TjIA+YAAAAAAAAAAAAAAAAAAFCYp3olpfHg0bieyhMU/0S0vjwaJwG8E30I+w3512nyhurxFgN3yP8N+ddp8obq8QGGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyTBev8AeOy+cKVf/maWSs68CEZ4NHtbsgsxzlaxra+mVznKjWoiTNvVXLuIhXC2vS8ZpPnRd4hqNo37tquIonSNGs8ah+VZFF+leq//AFpO0nIoHGZrYZbKo0jlhlVK1VVscjJFRPF5N1Uau4hPwhdwq6rlmKq/MABK0AAAAAAAA5afdqbu6h8W759kduhMNtbGsFNLV0FPUvqqtj54WyuYxIshquTeS9t9x6G0xRccruiHumC2bhHtWmp44IpYEhhYkcaLAxzkY3evcu+p+jbWtj82m/h2Fabd/q6SjL2R4Y1tzr/X7vZ2bYMaaz7Mnq46mqlfDkKjJEjyFypGtW/Jbf8AeasVTLLdwg2nW0r6aeSB0EtyPa2FjHLkuRyXOTe3UQxJTNRFUR8Xm1Gfcx67muPHhp6PmpwcqcHtRAAAAAAAAAAAAAAAADe+LDWrFS2giNa7KmhXdvS66N3AaINoYFtltDZsNU2slfC6WSNzMmN8iKjWORd1ibm6qGK94/B8Hm2Oy6LFeREX9PDz8+UKJS2X/gZ0qSphjmWTZNaL1REV07VuTe/wI0NxrhSsTjUn8PN3TRez+0oqy2KupgVXwTSo+NytViq1I2N3Wu3U3UUxY83ZmfG2G2bWDbt07tprrz0nX5fq8IAFpzwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIc3+44AHKr7jgAAAAAAAAAAAAOWLun1V6HxAH2R6HGUh8gB9stBlIfEAAAAAAAAAAAAAAAAAAAAOzVQ6gD6XodHHAAAAAAAAAAAAAAAAAA7SMVrlauSqtVWqrXI9t6Lctz2qqOT9UW46gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOUTd/8AETpXeOAB/9k=\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"600\"\n",
       "            height=\"400\"\n",
       "            src=\"https://www.youtube.com/embed/TmPfTpjtdgg\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x7f8ab46542e8>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#If no video appears here - run this cell by selecting it and clicking the run button above or alternatively\n",
    "# by pressing shift-enter\n",
    "\n",
    "#title DQN Going Superhuman\n",
    "#@markdown In this now classic video, watch & see how the DQN algorithm learns the absolute best way to play the Atari game of Breakout.\n",
    "from IPython.display import YouTubeVideo \n",
    "YouTubeVideo('TmPfTpjtdgg', width=600, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "Lets take a look at the DQN algorithm, training in 2015..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2MBERISGBUYLxoaL2NCOEJjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY//AABEIAWgB4AMBIgACEQEDEQH/xAAbAAEAAgMBAQAAAAAAAAAAAAAABQYDBAcBAv/EAEgQAQABAwICAwwIBAMFCQAAAAABAgMEBREGEhMhMRUyNVFxcoKSsbLB0RYiNEFTVGGBFJGhogdScyMkVZPwFyUzNkJiY4Th/8QAGAEBAAMBAAAAAAAAAAAAAAAAAAECAwT/xAAjEQEAAgIBAwQDAAAAAAAAAAAAAQIDERITIVEEFCIxMkFC/9oADAMBAAIRAxEAPwDn4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ2NEx50f8AjOkuc/Rc+28bb7eRBLhR/wCWP/rT7FPBYe4mDb0+jKyL96mmaKaqttp232/T9X1iaJpubbm5j5N+qmJ2meqOv+SWtRd7k2egpoqudHRtFfZ9zJhRkRbq/iaLVFW/VFrs2BVNO061l6pcxa6q4oo5tpjbfqlvZOlaVi36bN7Kv03KoiYjbft/ZH0ZtzA1S/et001Vc1UbVeVuUcQ37t6imrHsTvMRvtIGsaNY0/Ei9auXKqprinaqY29iDW3ijwZT/qR7JVIBlx4tVX6Iv1TRamfrVR2xDEy49ivJv0Wbe3NXO0bgk/4fQ/zuR6v/AONjM0PEtadXlWbt2rajmp322n+jJp+m6lgRXFFrGr59u/mZ2SWq83cS9zxEVdHG8U9kSClPHrwBI6LgW9Qya7d2qumKaOb6vlRyc4V+33f9P4wDQ1bDowc2bFuqqqmIid6u1pJbiXwtV5tKJAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAbmk41vL1G1Yu78lW++07dkTLTSPD/hrH9L3ZBPfR3A8Vz1z6O4HiueulwER9HcDxXPXPo7geK566XARH0dwPFc9c+juB4rnrpcBF9wcTl5ea/wAvZt0k7Pj6O4HiueulwEZGh4sRtFd+Ij/5JO4mN+Jkf82UmAiZ4dwJneYu7+efR3A8Vz10sAi6tBw642qqvVR4puTL4+juB4rnrpcBEfR3A8Vz13scPYNM7x0sTH3xWlgEZ3ExvxMj/myToWJVG1Vd+Yn7puSkwER9HcDxXPXPo7geK566XARH0dwPFc9d9U8P4VE70Tepn9K9kqAiquH8Kud6ulmfHNb5+juB4rnrpcBEfR3A8Vz10Tr+m4+BTZmxFX15nfmnfxLarvFveY3lq+AK2AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkeH/DWP6Xuyjkjw/wCGsf0vdkF1aFWtafTVNM5ERMTtP1Z+Tfc9yPtF3z59oLl3b078zHqz8ju3p35mPVn5KSAu3dvTvzMerPyO7enfmY9WfkpIC7d29O/Mx6s/I7t6d+Zj1Z+SkgLt3b078zHqz8ju3p35mPVn5KSAu3dvTvzMerPyO7enfmY9WfkpIC7d29O/Mx6s/I7t6d+Zj1Z+SkgLt3b078zHqz8ju3p35mPVn5KSAu3dvTvzMerPyO7enfmY9WfkpIC7d29O/Mx6s/I7t6d+Zj1Z+SkgLt3b078zHqz8ju3p35mPVn5KSAu3dvTvzMerPyO7enfmY9WfkpIDoVm7RftU3bVXNRVG8T40Dxb3mN5avglNF8E43mIvi3vMby1fAFbAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASPD/hrH9L3ZRyR4f8NY/pe7ILq57kfaLvnz7XQVCz7NVjNvW69uaKp7P5g1wAAAAAAAAAAAAAAAAAAAAAXjRfBON5iL4t7zG8tXwSmi+CcbzEXxb3mN5avgCtgAAAAAAAAAvN/hiuvibArsabvp002puzEfVnq+tuh7ugWqrl/Jyc2zgY1eTctWIqpmrm2qmOyOyI8bZy9aw7nF2n5tvJmcS1FrpKuWrq5Y6+rbcyr+ma1h27FzUaMSvGyL1cVXKKpi5brr5t42jt/QEVk6Dk4lvUZv10U1YE24qiOvn556piXlWiXKa9Kp6aj/vHbk6p+pvVy9abv6tp+rZGsY9WVGLayqbNNi7cpnaej8e3Zu183UdOpzOH6cfL6W1g8sXrnR1RttXEzO2wNPN0C3iVVWI1LHu5tNyKP4emJ3652j63Zv19cMeqaNj6dReo7qWbuXYmIuWIoqjr369pnqnZq5l23ka3fvUXpotXMiqqLsRO8UzVvzbdv6rDqmo41zRMm1m6lj6nkV8v8NVRYmm5R19c1VbR933AqKZwdCov4NnLzNQs4dvIqmixFdM1TXMTtM9XZG/3oZYrVWnaromn42TqFOHdwZriuK6Jnnoqq5t6do7f0B9aFoVmniaMHVLtum5Zu7TYmJmLvVM9Ux93ZPW0eIrtm7mUdDdxa6aaZj/drE2op6+yYntlIUa1h3OOo1SuubeJFffTTMztFHLE7R1q5dmKrtdUdk1TMAy4GTRiZMXrmNayaYiY6O7EzTKzazl4WJpmn3rejYPPm2Kq6p5Z+pO+3V1qimdbzsbK0vR7Ni5z3MexNF2OWY5Z3/Xt/YGSzw9b7n4ubmanYxbOTTM0c1M1VbxO220e1ixNHw7+1NzWca1druTRboiiqrfadomZiOqJeatm4+To2kY9m5zXce3XTdp2mOWZq3jy/smtN1TFtaVgRY1SnT6bG/8AFWotTNd2ebtiYjr3j+QI3RNJmjiKcPJu49GTYuctNm9RNdF2evxfd5Wpp2k16pqGXYpvWrHQ0V3aqquqmIpnr8nambOdpdfGV7V69Sot2Kb/ADUUzauTNcbdsbU9X7tPFv6fgZWpVU59F+nKwr1FE0W64+vVMbU9cf17AamfotGJZxcq1nWsjEyK5o6amiY5Jjt3ietl4hxIxMbTeSrHuW67EzRds25pmuN+2rf7yMrDv8OYWn15VNq7TlV11zVRVMUUzHb1R1/s2dcr0zJ0vBt4+q2rl3CsTb5Is3I6Sd9+qZp2j9wLvCU27t3GjUbFeZTa6WmxFM71U7b9vZE/owYfDnT4+LXkZ9nGvZk/7varpmZrjfbeZjs3+5IRrWn/AEzr1D+I/wB1mxyRXyVdvR7dm2/a9wbmBqtehVXcqqxk4vJa6Doqqul5auqaZiNvLuCs6hiVYGffxK6oqqs1zRNUdk7NdJcR1RXxDqFVM7x/EV9f7o0BI8P+Gsf0vdlHJHh/w1j+l7sguiqa5kY9/HszRNucjmnpOWNp/dbHPcj7Rd8+faDNZwMm/bi5bt81M/fvD77lZn4P9Yb+mZ2NYwqbd25y1RM9XLLb7qYf439s/JSZl2UxY5rEzZCVaZl00zVNrqiN564aiy1ahi3aZt0Xd6645aY5Z65lFdxM/wDA/vp+aYt5Z5McR+HdHM+Ph38mmZs0c0R1T1truJn/AIH99PzSmkYOTi27kXrfLNUxMfWifYTeIhGPFNralD9ysz8H+6DuVmfg/wB0LP0dfiOjr8SnUdXtaeVNqpmiqaauqYnaXylL2jZ1d6uqmxvE1TMfXp+b47iZ/wCB/fT82nKPLhmlt/TUx8e7k1TTap5piN562fuVmfg/3QktJ03Kxr9dV21yxNO0fWifv8qV6OvxKTkiJdWL08Wruysdysz8H+sNW7brs3Jt3I2qp7YXHo6/Eg87Sc29mXLluzvTVPVPNHzK5IlXNgikbqh2SzarvXIt243qnshu9xM/8D++n5tjT9KzLGbbuXLO1NO+880eLyrTaPLCuOZmImGn3KzPwf6wdysz8H+sLP0dfiOjr8TPqu32tPKoX7FzHrii7Ty1TG+27EntV03LycmK7VrmpimI35oj4ozJ07KxLfSX7XLTvtvzRPsaRaJceTHNZmGtTTNdUU09czO0NvuVmfg/3Q1rFUU37dVXVEVRMrH3Uw/xv7Z+SbTMfS+KlLb5TpJ6VbqtabYt1xtVTTtMIni3vMby1fBN4dym7i266J3pqjeJQnFveY3lq+AwtEROoVsBKAAAAAAAAAAAAAAAAAAAAAAAAAABI4mu6nhYsY2Nl127Ub7RER1b9u07bwjgHszNUzMzMzPbMvAASPD/AIax/S92UckeH/DWP6XuyC6ue5H2i758+10Jz3I+0XfPn2g+B429PwL2oXqrdjl5qaeaeadiZ13kY8H7dj/6lPtXfZAY3DmdaybVyqbXLRXFU7VeKVl6Kpx58lZ1qXThtx2w7GzN0VX/AFJ0VX/Uufm26sMOxszdFUdFUczqww7GzRua1hW7lVFVyqKqZmJ+rPa3cO9Rm2OmsTvRvtvPUvaLVjcnVh7sbPnNv28GzF2/O1Ezt1Rv1tO1rWFdu0W6K6pqrmKY+rPbJEWtG4OrDe2Nmboqkbk6tiY1+uzdrqiuidpjllFd2+jqw3NjZH93cD8Sr1ZZcbVcTKyKLFmuZrr7ImmYXml4/R1YbezQr1jBorqpqvTE0ztP1ZSvRVKFmdWZf/1KvathiL72pfNMfS5Y1+1lWuks1c1G+2+2yP4kjbTqf9SPZLNw3bmrSomP88vjieiadNpmfxI9kkTEZeKLZOVO6qDwdrkXjRvBON5iL4t7zG8tXwSmi+CcbzEXxb3mN5avgCtgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJHh/w1j+l7so5I8P+Gsf0vdkF1c9yPtF3z59roTnuR9ou+fPtBjSmg6hZ07KruX4qmmqjl+rG/wB6LEWrFo1KV0tcSYV27Rbppu81cxTG9Mff+6V6WnxS53jXIs5Nq5VvMUVxVO36Ssn0lxfwr38o+bkv6aP5Z3m/8rB0tPik6WnxSr/0lxfwr38o+Z9JcX8K9/KPmp7aWfLL4TeXnWsTHqv3Iq5adt9o/XZG/SjA/wAt71Y+aM1LXLGZgXLFFu5FVW20zEbdsT40C0p6auvk1pNpj5MmRci7kXblO+1Vc1Rv+srbwxXFOkxExPfypqc0nWbGDhxZuUXJq5pnenbZtlpyrovNoj4pTiquKtMpiN//ABI9kqri3Is5dm7Vvy0V01Tt+kpvNz7esWP4ezTXRVTPPvXEbf8AXWjq9KuU0TV0lPVG7TBgtFO0FZnXyWL6UYH+W96sfNWNTyaMvUL1+3ExTXO8b9vY1XilMVaTuF3rb0rJow9Rs37sTNFEzvt29kw0xpMbjSFy+lGB/lverHzVLIuRdyLtynsqrmqN/wBZYhnjxVx/ilYtE1vFwMCLF6m5NUVTP1Y3h867rWNqGFTZs03IqiuKvrRt41fEdGvLn+wAaoXjRvBON5iL4t7zG8tXwSmi+CcbzEXxb3mN5avgCtgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJHh/w1j+l7so5I8P8AhrH9L3ZBdXPcj7Rd8+fa6E57kfaLvnz7QYwAevAAASACAevAG1g5NONdmuqmZiY26m5c1W1Vbqp6OrriYRI2rmvWOMGnrwGQAIAAAAAAF40XwTjeYi+Le8xvLV8EpovgnG8xF8W95jeWr4ArYAAAAAAAAAAAAAAAC62f8Ocu7ZouRn2YiumKtuSfvUpsxqGbEREZmRER90XavmC3/wDZrl/8QsepKN1/g69oen/xd3MtXI54pimmmYmd2vw1Gdquu42LOXkTb5ua5/tKu9jrn7/2/dMf4l6l0ufj6dbq+pYp564j/NPZ/KPaCK4c4Tv8QYl3ItZVuzFuvkmKqZnfq3+KX/7Ncv8A4hY9SW9/h11cOaj/AKlXuQos5mpRbi5OTlRRPZVz1bSCQ4l4ZvcPRjTdyKL3T823LTMbbbfNBst7Jv5G3T3rl3l7OeqZ2/mxAAAAAAAAAJHh/wANY/pe7KOSPD/hrH9L3ZBdXPcj7Rd8+fa6E57kfaLvnz7QY3rx6DwAAAAAAAAAAAAAAAAAAAF40XwTjeYi+Le8xvLV8Epo3gnG8xF8W95jeWr4ArYAAAAAAAAAAAAAAAAPuzarv3qLNqnmuXKoppjxzIL/AP4dYVGHpuZrGRHLTMTTTVP+Snrmf5+xRtSzK9Q1HIy7m/NeuTV1/dH3R/J0HjC7RoXCGPpVirau7EW+rtmI66p/efa5qC1cL8V2NA0q/Yqxq7965d54jeIp22iOuUvi/wCI9u5di3madyWZ6pqor5tv2mOtz4BfeN+H8OrTqdb0ymmmidqrkW4+rVTV2VR/NQnT8imrE/wx5MremqcaIiKu2N56o/rDmAAAAAAAAACR4f8ADWP6Xuyjkjw/4ax/S92QXVz3I+0XfPn2uhOe5H2i758+0GMAAAAAAAAAAAAAAAAAAAAAF40XwTjeYi+Le8xvLV8EpovgnG8xF8W95jeWr4ArYAAAAAAAAAAAAAAACZ4TysDB1y3l6lXNNqzTNVO1M1b1dkdnlmf2QwCf4y1u3resdLj1VTjWqIot7xtv98zt5fYgABddB1Hhi9oONgazZiL9rmjnqtz99Uz1VU9f3t+zVwJp9yMi3NNyunrpieevr8k9TnYCzcW8WVa7y4+NRVaw6J32q7a58c/JWQAAAAAAAAASPD/hrH9L3ZRyR4f8NY/pe7ILq57kfaLvnz7XQnPcj7Rd8+faDGAAAAAAAAAAAAAAAAAAAAAC8aN4JxvMRfFveY3lq+CU0XwTjeYi+Le8xvLV8AVsAAAAAAAAAAAAAAABaL2gZGfw9o97T8W3Nc0XOmriqmiap5+rfeY3VdK6nnWMnRdIxrVUzdxqLkXI222ma94/oCZ1zQcadX1HIuXaMDAxpt0f7O1zb1VUx1RTG3lR9zhqab1XR5dNzHnCrzLV2Le3PTTHZtv1SktR1rS9VualiXciuxYv3Ld21f6KatqqaYpmJp7fufFGuabRm4+JFd2cC1g14dV/k2mZrjrq5fECAq06adEo1Lpeqq/Nno+X/wBu++6S1Lh/B0yqq1k6xEZHRxcot048zvvG8RM79T51LJ023w7a03Cyq8i5RlTdqrm3NETE07dW7X4mzrGo6xXkY1U1W5t0UxMxt1xTESDNl6Fi4OJbqy9Rqt5F2xF6i3GPNVM7xvFPPv2/sg1vwdXwcLBqt3dWvZuNNiaYwbmPPVVMdnNPVERPiVAAAAAAAAABI8P+Gsf0vdlHJHh/w1j+l7sgurnuR9ou+fPtdCc9yPtF3z59oMYAAAAAAAAAAAAAD14AAAAAAAvGi+CcbzEXxb3mN5avglNF8E43mIvi3vMby1fAFbAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASPD/hrH9L3ZRyR4f8ADWP6XuyC6ue5H2i758+10Jz3I+0XfPn2gxgAAAAAAAAAAAAAAAAAAAAAvGi+CcbzEXxb3mN5avglNF8E43mIvi3vMby1fAFbAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASPD/AIax/S92UckeH/DWP6XuyC6ue5H2i758+10Jz3I+0XfPn2gxgAAAAAAAAAAAAAAAAAAAAAvGi+CcbzEXxb3mN5avglNF8E43mIvi3vMby1fAFbAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASPD/hrH9L3ZRyS4e8NY/pe7ILo57kfaLvnz7XQtpc+yIn+IuefPtCO7EPdpBLwAQAAAAAAAAAAAAAAAAA9Bd9F8E43mIvi3vMby1fBKaL4JxvNRfFveY3lq+AnUq2AIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEjw/4Zx/S92Uc+7N25YuxctVTRXHZMCLRuJh0FX7mq4lNyqJ5t4nbvUR3Vz/zVz+bTmZqmZmd5lExs9Pyw7WGdWw9p771VeHhEaa3yTf7evASzAAAAAAAAAAAAAAAAAegtGmzPc+z1/8ApaHEM702Ov75+CHiuqI6pl5NUz2zurx77dNs+6cdPAFnMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//Z\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"600\"\n",
       "            height=\"400\"\n",
       "            src=\"https://www.youtube.com/embed/eG1Ed8PTJ18\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x7f8ab4649160>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#If no video appears here - run this cell by selecting it and clicking the run button above or alternatively\n",
    "# by pressing shift-enter\n",
    "\n",
    "\n",
    "#@title DQN Training...\n",
    "#@markdown Here we can see a DQN Agent training\n",
    "from IPython.display import YouTubeVideo \n",
    "YouTubeVideo('eG1Ed8PTJ18', width=600, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing you might have noticed in the video above is \"hours training\". Unfortunately back in 2015, DQN took 75 hours to train a DQN up to human level.\n",
    "\n",
    "**That is 3.125 days worth of compute time on a normal 2015 computer!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things have changed since 2015 however & lucky for us the introduction of DQN kicked off a wave of resurgence in academic study focussing on Reinforcement Learning, [leading to many new algorithms](https://arxiv.org/pdf/1708.07902.pdf), some of which we can see in the image below! \n",
    "\n",
    "**Notice how DQN is the middle!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](https://i.ibb.co/sy9xRF2/Screen-Shot-2019-02-21-at-9-30-09-am.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In todays lab we are going to making use of an algorithm developed by Deepmind called \"IMPALA\" which is short hand for[ \"Importance Weighted Actor-Learner Architectures\"](https://arxiv.org/abs/1802.01561). IMPALA is a distant couzin of DQN and is state of the art being [actively used by Deepmind right now for their research. ](https://www.quora.com/Is-Deepmind-AIs-alphastar-a-model-free-or-model-based-RL-model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike DQN, IMPALA allows us to parallelize training of our Reinforcement Learning agents and makes use of both CPUs & GPUs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We are going to show you how, using AWS, you can parallelize the training of your reinforcment learning algorithms to get insanely fast turn around times & results for your reinforcement learning experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective of today's lab is:\n",
    "\n",
    "1. Make use of a state of the art Reinforcement Learning algorithm (IMPALA) to learn how to play the Atari game of breakout!\n",
    "2. Make use of Amazon's state of the art compute resources. Instead of 3.125 days of training time, we are going to train our RL Agent to play Atari Breakout in 15 minutes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In the first cell block that you ran at the start of this workshop we actually started training out IMPALA algorithm on the Atari game of breakout!**\n",
    "\n",
    "So you should be roughly 8 minutes though the training process already!\n",
    "\n",
    "We are now going to walk through exactly what you did- so you can train your own experiments in the future!\n",
    "\n",
    "But first lets see how our Reinforcement Learning agent doing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizating our Reinforcement Learning Algorithm Playing!\n",
    "\n",
    "While it's running there are a variety of ways we can track progress of the running training job.  Some intermediate output gets saved to S3 during training, so we'll set up to capture that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Job name: {}\".format(job_name))\n",
    "\n",
    "s3_url = \"s3://{}/{}\".format(s3_bucket,job_name)\n",
    "\n",
    "output_tar_key = \"{}/output/output.tar.gz\".format(job_name)\n",
    "\n",
    "intermediate_folder_key = \"{}/output/intermediate/\".format(job_name)\n",
    "output_url = \"s3://{}/{}\".format(s3_bucket, output_tar_key)\n",
    "intermediate_url = \"s3://{}/{}\".format(s3_bucket, intermediate_folder_key)\n",
    "\n",
    "print(\"S3 job path: {}\".format(s3_url))\n",
    "print(\"Output.tar.gz location: {}\".format(output_url))\n",
    "print(\"Intermediate folder path: {}\".format(intermediate_url))\n",
    "    \n",
    "tmp_dir = \"/tmp/{}\".format(job_name)\n",
    "os.system(\"mkdir {}\".format(tmp_dir))\n",
    "print(\"Create local folder {}\".format(tmp_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Watch your Reinforcement Agent training live!!\n",
    "\n",
    "Videos of certain rollouts get written to S3 during training.  We can observe our Reinforcement learning agent learning live!  Here we fetch the last 10 videos from S3, and render the last one.  \n",
    "\n",
    "#### Initially you'll notice that the agent will lose the game straight away because it's not very intelligent yet, but if you continue to run the cell below every few minutes throughout the workshop you'll see the agent \"learn\" and get more intelligent over time.\n",
    "&nbsp;\n",
    "&nbsp;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "recent_videos = wait_for_s3_object(s3_bucket, intermediate_folder_key, tmp_dir, \n",
    "                                fetch_only=(lambda obj: obj.key.endswith(\".mp4\") and obj.size>0), limit=10)\n",
    "\n",
    "last_video = recent_videos[-1]  # Pick which video to watch\n",
    "os.system(\"sudo mkdir -p ./src/tmp_render/ && sudo cp {} ./src/tmp_render/last_video.mp4\".format(last_video))\n",
    "HTML('<video src=\"./src/tmp_render/last_video.mp4\" controls autoplay></video>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot metrics for training job\n",
    "We can see the reward metric of the training as it's running, using algorithm metrics that are recorded in CloudWatch metrics.  We can plot this to see the performance of the model over time.\n",
    "\n",
    "#### Rerun the cell below every few minutes throughout the workshop to see the plot of the agent's \"average reward\" over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from sagemaker.analytics import TrainingJobAnalytics\n",
    "\n",
    "df = TrainingJobAnalytics(job_name, ['episode_reward_mean']).dataframe()\n",
    "num_metrics = len(df)\n",
    "if num_metrics == 0:\n",
    "    print(\"No algorithm metrics found in CloudWatch\")\n",
    "else:\n",
    "    plt = df.plot(x='timestamp', y='value', figsize=(12,5), legend=True, style='b-')\n",
    "    plt.set_ylabel('Mean reward per episode')\n",
    "    plt.set_xlabel('Training time (s)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of lab."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Sagemaker RL Lab - Summit 2019 - One Click.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "conda_tensorflow_p36",
   "language": "python",
   "name": "conda_tensorflow_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}