t81_558_class_06_2_cnn.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3.9 (tensorflow)",
      "language": "python",
      "name": "tensorflow"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "colab": {
      "name": "Copy of t81_558_class_06_2_cnn.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O8hkxXtRCzJj"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_06_2_cnn.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YDTXd8-Lmp8Q"
      },
      "source": [
        "# T81-558: Applications of Deep Neural Networks\n",
        "**Module 6: Convolutional Neural Networks (CNN) for Computer Vision**\n",
        "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
        "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ncNrAEpzmp8S"
      },
      "source": [
        "# Module 6 Material\n",
        "\n",
        "* Part 6.1: Image Processing in Python [[Video]](https://www.youtube.com/watch?v=4Bh3gqHkIgc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_06_1_python_images.ipynb)\n",
        "* **Part 6.2: Keras Neural Networks for Digits and Fashion MNIST** [[Video]](https://www.youtube.com/watch?v=-SA8BmGvWYE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_06_2_cnn.ipynb)\n",
        "* Part 6.3: Implementing a ResNet in Keras [[Video]](https://www.youtube.com/watch?v=qMFKsMeE6fM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_06_3_resnet.ipynb)\n",
        "* Part 6.4: Using Your Own Images with Keras [[Video]](https://www.youtube.com/watch?v=VcFja1fUNSk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_06_4_keras_images.ipynb)\n",
        "* Part 6.5: Recognizing Multiple Images with YOLO Darknet [[Video]](https://www.youtube.com/watch?v=oQcAKvBFli8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_06_5_yolo.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6hj6qzYNCzJl"
      },
      "source": [
        "# Google CoLab Instructions\n",
        "\n",
        "The following code ensures that Google CoLab is running the correct version of TensorFlow."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fU9UhAxTmp8S",
        "outputId": "df36ba1b-bd27-4cba-8bfc-f587988fcea8"
      },
      "source": [
        "try:\n",
        "    %tensorflow_version 2.x\n",
        "    COLAB = True\n",
        "    print(\"Note: using Google CoLab\")\n",
        "except:\n",
        "    print(\"Note: not using Google CoLab\")\n",
        "    COLAB = False\n",
        "\n",
        "# Nicely formatted time string\n",
        "def hms_string(sec_elapsed):\n",
        "    h = int(sec_elapsed / (60 * 60))\n",
        "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
        "    s = sec_elapsed % 60\n",
        "    return f\"{h}:{m:>02}:{s:>05.2f}\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Note: using Google CoLab\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jf_otSJdmp8k"
      },
      "source": [
        "# Part 6.2: Keras Neural Networks for Digits and Fashion MNIST\n",
        "\n",
        "# Computer Vision\n",
        "\n",
        "This module will focus on computer vision.  There are some important differences and similarities with previous neural networks.\n",
        "\n",
        "* We will usually use classification, though regression is still an option.\n",
        "* The input to the neural network is now 3D (height, width, color)\n",
        "* Data are not transformed, no z-scores or dummy variables.\n",
        "* Processing time is much longer.\n",
        "* We now have different layer times: dense layers (just like before), convolution layers and max pooling layers.\n",
        "* Data will no longer arrive as CSV files. TensorFlow provides some utilities for going directly from image to the input for a neural network.\n",
        "\n",
        "\n",
        "# Computer Vision Data Sets\n",
        "\n",
        "There are many data sets for computer vision.  Two of the most popular classic datasets are the MNIST digits data set and the CIFAR image data sets. We will not use either of these datasets in this course, but it is important to be familiar with them since neural network texts often refer to them.\n",
        "\n",
        "The [MNIST Digits Data Set](http://yann.lecun.com/exdb/mnist/) is very popular in the neural network research community.  A sample of it can be seen in Figure 6.MNIST.\n",
        "\n",
        "**Figure 6.MNIST: MNIST Data Set**\n",
        "![MNIST Data Set](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_mnist.png \"MNIST Data Set\")\n",
        "\n",
        "[Fashion-MNIST](https://www.kaggle.com/zalando-research/fashionmnist) is a dataset of [Zalando](https://jobs.zalando.com/tech/)'s article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. Fashion-MNIST is intended to serve as a direct **drop-in replacement** for the original [MNIST dataset](http://yann.lecun.com/exdb/mnist/) for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits. This data can be seen in Figure 6.MNIST-FASHION.\n",
        "\n",
        "**Figure 6.MNIST-FASHION: MNIST Fashon Data Set**\n",
        "![mnist-fashion](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/mnist-fashion.png \"mnist-fashion\")\n",
        "\n",
        "The [CIFAR-10 and CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html) datasets are also frequently used by the neural network research community.\n",
        "\n",
        "**Figure 6.CIFAR: CIFAR Data Set**\n",
        "![CIFAR Data Set](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_cifar.png \"CIFAR Data Set\")\n",
        "\n",
        "The CIFAR-10 data set contains low-rez images that are divided into 10 classes.  The CIFAR-100 data set contains 100 classes in a hierarchy. \n",
        "\n",
        "# Convolutional Neural Networks (CNNs)\n",
        "\n",
        "The convolutional neural network (CNN) is a neural network technology that has profoundly impacted the area of computer vision (CV). Fukushima  (1980) [[Cite:fukushima1980neocognitron]](https://www.rctn.org/bruno/public/papers/Fukushima1980.pdf) introduced the original concept of a convolutional neural network, and   LeCun, Bottou, Bengio & Haffner (1998) [[Cite:lecun1995convolutional]](http://yann.lecun.com/exdb/publis/pdf/lecun-bengio-95a.pdf) greatly improved this work. From this research, Yan LeCun introduced the famous LeNet-5 neural network architecture.  This chapter follows the LeNet-5 style of convolutional neural network.  \n",
        "Although computer vision primarily uses CNNs, this technology has some application outside of the field. You need to realize that if you want to utilize CNNs on non-visual data, you must find a way to encode your data so that it can mimic the properties of visual data.  \n",
        "\n",
        "CNNs are somewhat similar to the self-organizing map (SOM) architecture that we examined in Chapter 2, “Self-Organizing Maps.”  The order of the vector elements is crucial to the training. In contrast, most neural networks that are not CNNs or SOMs treat their input data as a long vector of values, and the order that you arrange the incoming features in this vector is irrelevant. For these types of neural networks, you cannot change the order after you have trained the network. In other words, CNNs and SOMs do not follow the standard treatment of input vectors. \n",
        "The SOM network arranged the inputs into a grid.  This arrangement worked well with images because the pixels in closer proximity to each other are important to each other.  Obviously, the order of pixels in an image is significant.  The human body is a relevant example of this type of order. For the design of the face, we are accustomed to eyes being near to each other.  In the same way, neural network types like SOMs adhere to an order of pixels. Consequently, they have many applications to computer vision. \n",
        "\n",
        "This advance in CNNs is due to years of research on biological eyes. In other words, CNNs utilize overlapping fields of input to simulate features of biological eyes. Until this breakthrough, AI had been unable to reproduce the capabilities of biological vision.\n",
        "Scale, rotation, and noise have presented challenges in the past for AI computer vision research.  You can observe the complexity of biological eyes in the example that follows. A friend raises a sheet of paper with a large number written on it.  As your friend moves nearer to you, the number is still identifiable. In the same way, you can still identify the number when your friend rotates the paper.  Lastly, your friend creates noise by drawing lines on top of the page, but you can still identify the number. As you can see, these examples demonstrate the high function of the biological eye and allow you to understand better the research breakthrough of CNNs. That is, this neural network has the ability to process scale, rotation, and noise in the field of computer vision. This network structure can be seen in Figure 6.LENET.\n",
        "\n",
        "**Figure 6.LENET: A LeNET-5 Network (LeCun, 1998)**\n",
        "![A LeNET-5 Network](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_lenet5.png \"A LeNET-5 Network\")\n",
        "\n",
        "So far we have only seen one layer type (dense layers).  By the end of this course we will have seen:\n",
        "\n",
        "* **Dense Layers** - Fully connected layers.  (introduced previously)\n",
        "* **Convolution Layers** - Used to scan across images. (introduced this class)\n",
        "* **Max Pooling Layers** - Used to downsample images. (introduced this class)\n",
        "* **Dropout Layer** - Used to add regularization. (introduced next class)\n",
        "\n",
        "## Convolution Layers\n",
        "\n",
        "The first layer that we will examine is the convolutional layer. We will begin by looking at the hyper-parameters that you must specify for a convolutional layer in most neural network frameworks that support the CNN:\n",
        "\n",
        "* Number of filters\n",
        "* Filter Size\n",
        "* Stride\n",
        "* Padding\n",
        "* Activation Function/Non-Linearity\n",
        "\n",
        "The primary purpose for a convolutional layer is to detect features such as edges, lines, blobs of color, and other visual elements. The filters can detect these features. The more filters that we give to a convolutional layer, the more features it can detect.\n",
        "\n",
        "A filter is a square-shaped object that scans over the image. A grid can represent the individual pixels of a grid. You can think of the convolutional layer as a smaller grid that sweeps left to right over each row of the image. There is also a hyper parameter that specifies both the width and height of the square-shaped filter. The following figure shows this configuration in which you see the six convolutional filters sweeping over the image grid:\n",
        "\n",
        "A convolutional layer has weights between it and the previous layer or image grid. Each pixel on each convolutional layer is a weight. Therefore, the number of weights between a convolutional layer and its predecessor layer or image field is the following:\n",
        "\n",
        "```\n",
        "[FilterSize] * [FilterSize] * [# of Filters]\n",
        "```\n",
        "\n",
        "For example, if the filter size were 5 (5x5) for 10 filters, there would be 250 weights.\n",
        "\n",
        "You need to understand how the convolutional filters sweep across the previous layer’s output or image grid. Figure 6.CNN illustrates the sweep:\n",
        "\n",
        "**Figure 6.CNN: Convolutional Neural Network**\n",
        "![Convolutional Neural Network](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_cnn_grid.png \"Convolutional Neural Network\")\n",
        "\n",
        "The above figure shows a convolutional filter with a size of 4 and a padding size of 1. The padding size is responsible for the boarder of zeros in the area that the filter sweeps. Even though the image is actually 8x7, the extra padding provides a virtual image size of 9x8 for the filter to sweep across. The stride specifies the number of positions at which the convolutional filters will stop. The convolutional filters move to the right, advancing by the number of cells specified in the stride. Once the far right is reached, the convolutional filter moves back to the far left, then it moves down by the stride amount and\n",
        "continues to the right again.\n",
        "\n",
        "Some constraints exist in relation to the size of the stride. Obviously, the stride cannot be 0. The convolutional filter would never move if the stride were set to 0. Furthermore, neither the stride, nor the convolutional filter size can be larger than the previous grid. There are additional constraints on the stride (s), padding (p) and the filter width (f) for an image of width (w). Specifically, the convolutional filter must be able to start at the far left or top boarder, move a certain number of strides, and land on the far right or bottom boarder. The following equation shows the number of steps a convolutional operator\n",
        "must take to cross the image:\n",
        "\n",
        "$$ steps = \\frac{w - f + 2p}{s+1} $$\n",
        "\n",
        "The number of steps must be an integer. In other words, it cannot have decimal places. The purpose of the padding (p) is to be adjusted to make this equation become an integer value.\n",
        "\n",
        "## Max Pooling Layers\n",
        "\n",
        "Max-pool layers downsample a 3D box to a new one with smaller dimensions. Typically, you can always place a max-pool layer immediately following convolutional layer. The LENET shows the max-pool layer immediately after layers C1 and C3. These max-pool layers progressively decrease the size of the dimensions of the 3D boxes passing through them. This technique can avoid overfitting (Krizhevsky, Sutskever & Hinton, 2012).\n",
        "\n",
        "A pooling layer has the following hyper-parameters:\n",
        "\n",
        "* Spatial Extent (f)\n",
        "* Stride (s)\n",
        "\n",
        "Unlike convolutional layers, max-pool layers do not use padding. Additionally, max-pool layers have no weights, so training does not affect them. These layers simply downsample their 3D box input. The 3D box output by a max-pool layer will have a width equal to this equation:\n",
        "\n",
        "$$ w_2 = \\frac{w_1 - f}{s + 1} $$\n",
        "\n",
        "The height of the 3D box produced by the max-pool layer is calculated similarly with this equation:\n",
        "\n",
        "$$ h_2 = \\frac{h_1 - f}{s + 1} $$\n",
        "\n",
        "The depth of the 3D box produced by the max-pool layer is equal to the depth the 3D box received as input. The most common setting for the hyper-parameters of a max-pool layer are f=2 and s=2. The spatial extent (f) specifies that boxes of 2x2 will be scaled down to single pixels. Of these four pixels, the pixel with the maximum value will represent the 2x2 pixel in the new grid. Because squares of size 4 are replaced with size 1, 75% of the pixel information is lost. The following figure shows this transformation as a 6x6 grid becomes a 3x3:\n",
        "\n",
        "**Figure 6.MAXPOOL: Max Pooling Layer**\n",
        "![Max Pooling Layer](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_conv_maxpool.png \"Max Pooling Layer\")\n",
        "\n",
        "Of course, the above diagram shows each pixel as a single number. A grayscale image would have this characteristic. For an RGB image, we usually take the average of the three numbers to determine which pixel has the maximum value.\n",
        "\n",
        "# Regression Convolutional Neural Networks\n",
        "\n",
        "We will now look at two examples, one for regression and another for classification.  For supervised computer vision, your dataset will need some labels.  For classification, this label usually specifies what the image is a picture of.  For regression, this \"label\" is some numeric quantity the image should produce, such as a count. We will look at two different means of providing this label.\n",
        "\n",
        "The first example will show how to handle regression with convolution neural networks.  In this case we will provide an image and expect the neural network to count items in that image. We will use a [dataset](https://www.kaggle.com/jeffheaton/count-the-paperclips) that I created that contains a random number of paperclips. The following code will download this dataset for you.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GpfadrdQcVg8"
      },
      "source": [
        "import os\n",
        "\n",
        "DOWNLOAD_SOURCE = \"https://github.com/jeffheaton/data-mirror/releases/download/v1/paperclips.zip\"\n",
        "DOWNLOAD_NAME = DOWNLOAD_SOURCE[DOWNLOAD_SOURCE.rfind('/')+1:]\n",
        "\n",
        "if COLAB:\n",
        "  PATH = \"/content\"\n",
        "else:\n",
        "  # I used this locally on my machine, you may need different\n",
        "  PATH = \"/Users/jeff/temp\"\n",
        "\n",
        "EXTRACT_TARGET = os.path.join(PATH,\"clips\")\n",
        "SOURCE = os.path.join(EXTRACT_TARGET, \"paperclips\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ivCHAirHpNyT"
      },
      "source": [
        "Next, we download the images. This part depends on the origin of your images. The following code downloads images from a URL, where a ZIP file contains the images. The code unzips the ZIP file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CExT2Z6gpAhz",
        "outputId": "30c455eb-03aa-4a46-e582-7fd9ffefcddc"
      },
      "source": [
        "!wget -O {os.path.join(PATH,DOWNLOAD_NAME)} {DOWNLOAD_SOURCE}\n",
        "!mkdir -p {SOURCE}\n",
        "!mkdir -p {TARGET}\n",
        "!mkdir -p {EXTRACT_TARGET}\n",
        "!unzip -o -j -d {SOURCE} {os.path.join(PATH, DOWNLOAD_NAME)} >/dev/null"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-11-27 18:36:52--  https://github.com/jeffheaton/data-mirror/releases/download/v1/paperclips.zip\n",
            "Resolving github.com (github.com)... 192.30.255.112\n",
            "Connecting to github.com (github.com)|192.30.255.112|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/408419764/25830812-b9e6-4ddf-93b6-7932d9ef5982?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211127T183652Z&X-Amz-Expires=300&X-Amz-Signature=cb4a7f9e60c33c47995258a2c16d8129a1607db0399b900d191d442385852f66&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Dpaperclips.zip&response-content-type=application%2Foctet-stream [following]\n",
            "--2021-11-27 18:36:52--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/408419764/25830812-b9e6-4ddf-93b6-7932d9ef5982?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211127T183652Z&X-Amz-Expires=300&X-Amz-Signature=cb4a7f9e60c33c47995258a2c16d8129a1607db0399b900d191d442385852f66&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Dpaperclips.zip&response-content-type=application%2Foctet-stream\n",
            "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 163590691 (156M) [application/octet-stream]\n",
            "Saving to: ‘/content/paperclips.zip’\n",
            "\n",
            "/content/paperclips 100%[===================>] 156.01M  30.0MB/s    in 5.4s    \n",
            "\n",
            "2021-11-27 18:36:58 (29.0 MB/s) - ‘/content/paperclips.zip’ saved [163590691/163590691]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p4LWl8TzzI8a"
      },
      "source": [
        "For the regression, the labels are contained in a CSV file named **train.csv**.  This file has just two labels, **id** and **clip_count**.  The ID specifies the filename; for example, row id 1 corresponds to the file **clips-1.jpg**. The following code loads the labels for the training set and creates a new column, named **filename**, that contains the filename of each image, based on the **id** column."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w4W4LeGSqYya"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_csv(\n",
        "    os.path.join(SOURCE,\"train.csv\"), \n",
        "    na_values=['NA', '?'])\n",
        "\n",
        "df['filename']=\"clips-\"+df[\"id\"].astype(str)+\".jpg\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AGxjITNtz0sC"
      },
      "source": [
        "This results in the following dataframe."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "rJpaMpmfswIp",
        "outputId": "0ec161da-7e27-4d9f-b4ee-26c40e236cc4"
      },
      "source": [
        "df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>clip_count</th>\n",
              "      <th>filename</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>30001</td>\n",
              "      <td>11</td>\n",
              "      <td>clips-30001.jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>30002</td>\n",
              "      <td>2</td>\n",
              "      <td>clips-30002.jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>30003</td>\n",
              "      <td>26</td>\n",
              "      <td>clips-30003.jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>30004</td>\n",
              "      <td>41</td>\n",
              "      <td>clips-30004.jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>30005</td>\n",
              "      <td>49</td>\n",
              "      <td>clips-30005.jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19995</th>\n",
              "      <td>49996</td>\n",
              "      <td>35</td>\n",
              "      <td>clips-49996.jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19996</th>\n",
              "      <td>49997</td>\n",
              "      <td>54</td>\n",
              "      <td>clips-49997.jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19997</th>\n",
              "      <td>49998</td>\n",
              "      <td>72</td>\n",
              "      <td>clips-49998.jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19998</th>\n",
              "      <td>49999</td>\n",
              "      <td>24</td>\n",
              "      <td>clips-49999.jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19999</th>\n",
              "      <td>50000</td>\n",
              "      <td>35</td>\n",
              "      <td>clips-50000.jpg</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>20000 rows × 3 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "          id  clip_count         filename\n",
              "0      30001          11  clips-30001.jpg\n",
              "1      30002           2  clips-30002.jpg\n",
              "2      30003          26  clips-30003.jpg\n",
              "3      30004          41  clips-30004.jpg\n",
              "4      30005          49  clips-30005.jpg\n",
              "...      ...         ...              ...\n",
              "19995  49996          35  clips-49996.jpg\n",
              "19996  49997          54  clips-49997.jpg\n",
              "19997  49998          72  clips-49998.jpg\n",
              "19998  49999          24  clips-49999.jpg\n",
              "19999  50000          35  clips-50000.jpg\n",
              "\n",
              "[20000 rows x 3 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nN1AsQeysmGd"
      },
      "source": [
        "Separate into a training and validation (for early stopping)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CI4_qNaSqp31",
        "outputId": "9ba5822c-7f55-49ee-e7b1-a5b7d7433a78"
      },
      "source": [
        "TRAIN_PCT = 0.9\n",
        "TRAIN_CUT = int(len(df) * TRAIN_PCT)\n",
        "\n",
        "df_train = df[0:TRAIN_CUT]\n",
        "df_validate = df[TRAIN_CUT:]\n",
        "\n",
        "print(f\"Training size: {len(df_train)}\")\n",
        "print(f\"Validate size: {len(df_validate)}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training size: 18000\n",
            "Validate size: 2000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNoqk5uIz6FW"
      },
      "source": [
        "We are now ready to create two ImageDataGenerator objects.  We are currently using a generator, which creates additional training data by manipulating the source material.  This technique can produce considerably stronger neural networks. The generator below flips the images both vertically and horizontally. Keras will train the neuron network both on the original images as well as the flipped images.  This augmentation increases the size of the training data considerably. Module 6.4 goes deeper into the transformations you can perform.  You can also specify a target size to resize the images automatically.\n",
        "\n",
        "The function **flow_from_dataframe** loads the labels from a Pandas dataframe connected to our **train.csv** file.  When we demonstrate classification, we will use the flow_from_directory; which loads the labels from the directory structure rather than a CSV."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YZzeAHdfsy0O",
        "outputId": "7cb467b4-61e0-4ab8-95fa-fe79e83b4b16"
      },
      "source": [
        "import tensorflow as tf\n",
        "import keras_preprocessing\n",
        "from keras_preprocessing import image\n",
        "from keras_preprocessing.image import ImageDataGenerator\n",
        "\n",
        "training_datagen = ImageDataGenerator(\n",
        "  rescale = 1./255,\n",
        "  horizontal_flip=True,\n",
        "  vertical_flip=True,\n",
        "  fill_mode='nearest')\n",
        "\n",
        "train_generator = training_datagen.flow_from_dataframe(\n",
        "        dataframe=df_train,\n",
        "        directory=SOURCE,\n",
        "        x_col=\"filename\",\n",
        "        y_col=\"clip_count\",\n",
        "        target_size=(256, 256),\n",
        "        batch_size=32,\n",
        "        class_mode='other')\n",
        "\n",
        "validation_datagen = ImageDataGenerator(rescale = 1./255)\n",
        "\n",
        "val_generator = validation_datagen.flow_from_dataframe(\n",
        "        dataframe=df_validate,\n",
        "        directory=SOURCE,\n",
        "        x_col=\"filename\",\n",
        "        y_col=\"clip_count\",\n",
        "        target_size=(256, 256),\n",
        "        class_mode='other')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 18000 validated image filenames.\n",
            "Found 2000 validated image filenames.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_GVeYo6p2sdG"
      },
      "source": [
        "We can now train the neural network.  The code to build and train the neural network is not that different than in the previous modules.  We will make use of the Keras Sequential class to provide layers to the neural network.  We now have several new layer types that we did not previously see.\n",
        "\n",
        "* **Conv2D** - The convolution layers.\n",
        "* **MaxPooling2D** - The max pooling layers.\n",
        "* **Flatten** - Flatten the 2D (and higher) tensors to allow a Dense layer to process.\n",
        "* **Dense** - Dense layers, the same as demonstrated previously.  Dense layers often form the final output layers of the neural network.\n",
        "\n",
        "The training code is very similar to previously. This code is for regression, so a linear final activation is used, along with mean_squared_error for the loss function.  The generator provides both the *x* and *y* matrixes we previously supplied."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DBHlqCIgtWSq",
        "outputId": "cba1cdf7-9f2f-4a33-8049-655a35e25ac6"
      },
      "source": [
        "from tensorflow.keras.callbacks import EarlyStopping\n",
        "import time\n",
        "\n",
        "model = tf.keras.models.Sequential([\n",
        "    # Note the input shape is the desired size of the image 150x150 with 3 bytes color\n",
        "    # This is the first convolution\n",
        "    tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(256, 256, 3)),\n",
        "    tf.keras.layers.MaxPooling2D(2, 2),\n",
        "    # The second convolution\n",
        "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
        "    tf.keras.layers.MaxPooling2D(2,2),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    # 512 neuron hidden layer\n",
        "    tf.keras.layers.Dense(512, activation='relu'),\n",
        "    tf.keras.layers.Dense(1, activation='linear')\n",
        "])\n",
        "\n",
        "\n",
        "model.summary()\n",
        "epoch_steps = 250 # needed for 2.2\n",
        "validation_steps = len(df_validate)\n",
        "model.compile(loss = 'mean_squared_error', optimizer='adam')\n",
        "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto',\n",
        "        restore_best_weights=True)\n",
        "\n",
        "start_time = time.time()\n",
        "history = model.fit(train_generator,  \n",
        "  verbose = 1, \n",
        "  validation_data=val_generator, callbacks=[monitor], epochs=25)\n",
        "\n",
        "elapsed_time = time.time() - start_time\n",
        "print(\"Elapsed time: {}\".format(hms_string(elapsed_time)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " conv2d (Conv2D)             (None, 254, 254, 64)      1792      \n",
            "                                                                 \n",
            " max_pooling2d (MaxPooling2D  (None, 127, 127, 64)     0         \n",
            " )                                                               \n",
            "                                                                 \n",
            " conv2d_1 (Conv2D)           (None, 125, 125, 64)      36928     \n",
            "                                                                 \n",
            " max_pooling2d_1 (MaxPooling  (None, 62, 62, 64)       0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " flatten (Flatten)           (None, 246016)            0         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 512)               125960704 \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 1)                 513       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 125,999,937\n",
            "Trainable params: 125,999,937\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/25\n",
            "563/563 [==============================] - 58s 88ms/step - loss: 202.8745 - val_loss: 21.2090\n",
            "Epoch 2/25\n",
            "563/563 [==============================] - 48s 86ms/step - loss: 20.1897 - val_loss: 14.5325\n",
            "Epoch 3/25\n",
            "563/563 [==============================] - 50s 89ms/step - loss: 19.4199 - val_loss: 20.8415\n",
            "Epoch 4/25\n",
            "563/563 [==============================] - 51s 91ms/step - loss: 18.0697 - val_loss: 11.9485\n",
            "Epoch 5/25\n",
            "563/563 [==============================] - 51s 90ms/step - loss: 17.0723 - val_loss: 20.3418\n",
            "Epoch 6/25\n",
            "563/563 [==============================] - 49s 87ms/step - loss: 14.6164 - val_loss: 10.9961\n",
            "Epoch 7/25\n",
            "563/563 [==============================] - 49s 87ms/step - loss: 14.2032 - val_loss: 17.7755\n",
            "Epoch 8/25\n",
            "563/563 [==============================] - 49s 87ms/step - loss: 11.2403 - val_loss: 10.7504\n",
            "Epoch 9/25\n",
            "563/563 [==============================] - 48s 86ms/step - loss: 9.7663 - val_loss: 9.2796\n",
            "Epoch 10/25\n",
            "563/563 [==============================] - 49s 87ms/step - loss: 7.9981 - val_loss: 6.7420\n",
            "Epoch 11/25\n",
            "563/563 [==============================] - 48s 86ms/step - loss: 7.4416 - val_loss: 5.9866\n",
            "Epoch 12/25\n",
            "563/563 [==============================] - 49s 87ms/step - loss: 6.5176 - val_loss: 10.9515\n",
            "Epoch 13/25\n",
            "563/563 [==============================] - 49s 87ms/step - loss: 6.8314 - val_loss: 9.6828\n",
            "Epoch 14/25\n",
            "563/563 [==============================] - 49s 87ms/step - loss: 5.2195 - val_loss: 5.2810\n",
            "Epoch 15/25\n",
            "563/563 [==============================] - 48s 86ms/step - loss: 5.7389 - val_loss: 4.7575\n",
            "Epoch 16/25\n",
            "563/563 [==============================] - 49s 88ms/step - loss: 3.9572 - val_loss: 4.2037\n",
            "Epoch 17/25\n",
            "563/563 [==============================] - 48s 85ms/step - loss: 3.4524 - val_loss: 6.8304\n",
            "Epoch 18/25\n",
            "563/563 [==============================] - 48s 85ms/step - loss: 26.4173 - val_loss: 1445.4049\n",
            "Epoch 19/25\n",
            "563/563 [==============================] - 48s 85ms/step - loss: 43.6318 - val_loss: 17.2842\n",
            "Epoch 20/25\n",
            "563/563 [==============================] - 48s 85ms/step - loss: 14.1856 - val_loss: 11.6476\n",
            "Epoch 21/25\n",
            "563/563 [==============================] - ETA: 0s - loss: 10.6792Restoring model weights from the end of the best epoch: 16.\n",
            "563/563 [==============================] - 48s 85ms/step - loss: 10.6792 - val_loss: 9.2932\n",
            "Epoch 00021: early stopping\n",
            "Elapsed time: 0:17:17.00\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gNiELOX53PtU"
      },
      "source": [
        "This code will run very slow if you do not use a GPU. The above code takes approximately 13 minutes with a GPU.\n",
        "\n",
        "## Score Regression Image Data\n",
        "\n",
        "Scoring/predicting from a generator is a bit different than training.  We do not want augmented images, and we do not wish to have the dataset shuffled.  For scoring, we want a prediction for each input.  We construct the generator as follows:\n",
        "\n",
        "* shuffle=False\n",
        "* batch_size=1\n",
        "* class_mode=None\n",
        "\n",
        "We use a **batch_size** of 1 to guarantee that we do not run out of GPU memory if our prediction set is large.  You can increase this value for better performance.  The **class_mode** is None because there is no *y*, or label. After all, we are predicting."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QFXvtr-NtlQt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "59ad62c8-667f-4717-a0ea-c1419f9a3b14"
      },
      "source": [
        "df_test = pd.read_csv(\n",
        "    os.path.join(SOURCE,\"train.csv\"), \n",
        "    na_values=['NA', '?'])\n",
        "\n",
        "df_test['filename']=\"clips-\"+df_test[\"id\"].astype(str)+\".jpg\"\n",
        "\n",
        "test_datagen = ImageDataGenerator(rescale = 1./255)\n",
        "\n",
        "test_generator = validation_datagen.flow_from_dataframe(\n",
        "        dataframe=df_test,\n",
        "        directory=SOURCE,\n",
        "        x_col=\"filename\",\n",
        "        batch_size=1,\n",
        "        shuffle=False,\n",
        "        target_size=(256, 256),\n",
        "        class_mode=None)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 20000 validated image filenames.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TOuNIlpLAF5A"
      },
      "source": [
        "We need to reset the generator to ensure we are always at the beginning."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gS0CjJ4bt8jQ"
      },
      "source": [
        "test_generator.reset()\n",
        "pred = model.predict(test_generator,steps=len(df_test))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4n-t8k5bt_nG"
      },
      "source": [
        "df_submit = pd.DataFrame({'id':df_test['id'],'clip_count':pred.flatten()})\n",
        "df_submit.to_csv(os.path.join(PATH,\"submit.csv\"),index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sicFJd8u5c3v"
      },
      "source": [
        "# Classification Neural Networks\n",
        "\n",
        "Just like earlier in this module, we will load data. However, this time we will use a dataset of images of three different types of the iris flower. This zip file contains three different directories which specify the label of each image. The directories are named the same as the labels:\n",
        "\n",
        "* iris-setosa\n",
        "* iris-versicolour\n",
        "* iris-virginica\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zxeLaa1c5gGA"
      },
      "source": [
        "import os\n",
        "\n",
        "DOWNLOAD_SOURCE = \"https://github.com/jeffheaton/data-mirror/releases/download/v1/iris-image.zip\"\n",
        "DOWNLOAD_NAME = DOWNLOAD_SOURCE[DOWNLOAD_SOURCE.rfind('/')+1:]\n",
        "\n",
        "if COLAB:\n",
        "  PATH = \"/content\"\n",
        "  EXTRACT_TARGET = os.path.join(PATH,\"iris\")\n",
        "  SOURCE = EXTRACT_TARGET # In this case its the same, no subfolder\n",
        "else:\n",
        "  # I used this locally on my machine, you may need different\n",
        "  PATH = \"/Users/jeff/temp\"\n",
        "  EXTRACT_TARGET = os.path.join(PATH,\"iris\")\n",
        "  SOURCE = EXTRACT_TARGET # In this case its the same, no subfolder"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hset2s3P9MFV"
      },
      "source": [
        "Just as before we unzip the images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9sQdvl9F6Xru",
        "outputId": "0b653cea-3354-4d32-cc9f-1b55cda29e1d"
      },
      "source": [
        "!wget -O {os.path.join(PATH,DOWNLOAD_NAME)} {DOWNLOAD_SOURCE}\n",
        "!mkdir -p {SOURCE}\n",
        "!mkdir -p {TARGET}\n",
        "!mkdir -p {EXTRACT_TARGET}\n",
        "!unzip -o -d {EXTRACT_TARGET} {os.path.join(PATH, DOWNLOAD_NAME)} >/dev/null"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-11-27 18:55:29--  https://github.com/jeffheaton/data-mirror/releases/download/v1/iris-image.zip\n",
            "Resolving github.com (github.com)... 192.30.255.113\n",
            "Connecting to github.com (github.com)|192.30.255.113|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://github-releases.githubusercontent.com/408419764/d548babd-36c3-414e-add2-a5d9ab941e6e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211127T185529Z&X-Amz-Expires=300&X-Amz-Signature=7602fedc2bfbadef08788e5423e9df4e0723b38620b35128a6c5ef87163f7edf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Diris-image.zip&response-content-type=application%2Foctet-stream [following]\n",
            "--2021-11-27 18:55:29--  https://github-releases.githubusercontent.com/408419764/d548babd-36c3-414e-add2-a5d9ab941e6e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211127T185529Z&X-Amz-Expires=300&X-Amz-Signature=7602fedc2bfbadef08788e5423e9df4e0723b38620b35128a6c5ef87163f7edf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Diris-image.zip&response-content-type=application%2Foctet-stream\n",
            "Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.108.154, 185.199.109.154, 185.199.110.154, ...\n",
            "Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.108.154|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 5587253 (5.3M) [application/octet-stream]\n",
            "Saving to: ‘/content/iris-image.zip’\n",
            "\n",
            "/content/iris-image 100%[===================>]   5.33M  --.-KB/s    in 0.08s   \n",
            "\n",
            "2021-11-27 18:55:30 (63.6 MB/s) - ‘/content/iris-image.zip’ saved [5587253/5587253]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bS1p_9Mn9EPz"
      },
      "source": [
        "You can see these folders with the following command."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UrtRO9O-SBQc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7badfd48-6d53-4e55-c6ff-2227157d8e5e"
      },
      "source": [
        "!ls /content/iris"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "iris-setosa  iris-versicolour  iris-virginica\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cYDFKA8i9KMP"
      },
      "source": [
        "We setup the generator, similar to before.  This time we use flow_from_directory to get the labels from the directory structure."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u7EgpVqAdGvI",
        "outputId": "eb49248b-c7c3-4f1b-c1c8-f9d096b7eaed"
      },
      "source": [
        "import tensorflow as tf\n",
        "import keras_preprocessing\n",
        "from keras_preprocessing import image\n",
        "from keras_preprocessing.image import ImageDataGenerator\n",
        "\n",
        "training_datagen = ImageDataGenerator(\n",
        "  rescale = 1./255,\n",
        "  horizontal_flip=True,\n",
        "  vertical_flip=True,\n",
        "  width_shift_range=[-200,200],\n",
        "  #height_shift_range=0.5,\n",
        "  rotation_range=360,\n",
        "\n",
        "  fill_mode='nearest')\n",
        "\n",
        "train_generator = training_datagen.flow_from_directory(\n",
        "    directory=SOURCE, target_size=(256, 256), \n",
        "    class_mode='categorical', batch_size=32, shuffle=True)\n",
        "\n",
        "validation_datagen = ImageDataGenerator(rescale = 1./255)\n",
        "\n",
        "validation_generator = validation_datagen.flow_from_directory(\n",
        "    directory=SOURCE, target_size=(256, 256), \n",
        "    class_mode='categorical', batch_size=32, shuffle=True)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 421 images belonging to 3 classes.\n",
            "Found 421 images belonging to 3 classes.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AFesQtNZBZP0"
      },
      "source": [
        "Training the neural network with classification is similar to regression. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MBnwiM-XflQc",
        "outputId": "be0fc753-952b-480c-be7e-b00e3a8303c7"
      },
      "source": [
        "from tensorflow.keras.callbacks import EarlyStopping\n",
        "\n",
        "model = tf.keras.models.Sequential([\n",
        "    # Note the input shape is the desired size of the image 300x300 with 3 bytes color\n",
        "    # This is the first convolution\n",
        "    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(256, 256, 3)),\n",
        "    tf.keras.layers.MaxPooling2D(2, 2),\n",
        "    # The second convolution\n",
        "    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\n",
        "    tf.keras.layers.Dropout(0.5),\n",
        "    tf.keras.layers.MaxPooling2D(2,2),\n",
        "    # The third convolution\n",
        "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
        "    tf.keras.layers.Dropout(0.5),\n",
        "    tf.keras.layers.MaxPooling2D(2,2),\n",
        "    # The fourth convolution\n",
        "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
        "    tf.keras.layers.MaxPooling2D(2,2),\n",
        "    # The fifth convolution\n",
        "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
        "    tf.keras.layers.MaxPooling2D(2,2),\n",
        "    # Flatten the results to feed into a DNN\n",
        "    \n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dropout(0.5),\n",
        "    # 512 neuron hidden layer\n",
        "    tf.keras.layers.Dense(512, activation='relu'),\n",
        "    # Only 1 output neuron. It will contain a value from 0-1 where 0 for 1 class ('horses') and 1 for the other ('humans')\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "model.summary()\n",
        "\n",
        "model.compile(loss = 'binary_crossentropy', optimizer='adam')\n",
        "\n",
        "history = model.fit(train_generator, epochs=15, steps_per_epoch=10, \n",
        "                    verbose = 1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " conv2d_2 (Conv2D)           (None, 254, 254, 16)      448       \n",
            "                                                                 \n",
            " max_pooling2d_2 (MaxPooling  (None, 127, 127, 16)     0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " conv2d_3 (Conv2D)           (None, 125, 125, 32)      4640      \n",
            "                                                                 \n",
            " dropout (Dropout)           (None, 125, 125, 32)      0         \n",
            "                                                                 \n",
            " max_pooling2d_3 (MaxPooling  (None, 62, 62, 32)       0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " conv2d_4 (Conv2D)           (None, 60, 60, 64)        18496     \n",
            "                                                                 \n",
            " dropout_1 (Dropout)         (None, 60, 60, 64)        0         \n",
            "                                                                 \n",
            " max_pooling2d_4 (MaxPooling  (None, 30, 30, 64)       0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " conv2d_5 (Conv2D)           (None, 28, 28, 64)        36928     \n",
            "                                                                 \n",
            " max_pooling2d_5 (MaxPooling  (None, 14, 14, 64)       0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " conv2d_6 (Conv2D)           (None, 12, 12, 64)        36928     \n",
            "                                                                 \n",
            " max_pooling2d_6 (MaxPooling  (None, 6, 6, 64)         0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " flatten_1 (Flatten)         (None, 2304)              0         \n",
            "                                                                 \n",
            " dropout_2 (Dropout)         (None, 2304)              0         \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 512)               1180160   \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 1)                 513       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1,278,113\n",
            "Trainable params: 1,278,113\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/15\n",
            "10/10 [==============================] - 5s 423ms/step - loss: 0.7495\n",
            "Epoch 2/15\n",
            "10/10 [==============================] - 5s 474ms/step - loss: 0.6426\n",
            "Epoch 3/15\n",
            "10/10 [==============================] - 4s 435ms/step - loss: 0.6400\n",
            "Epoch 4/15\n",
            "10/10 [==============================] - 4s 422ms/step - loss: 0.6380\n",
            "Epoch 5/15\n",
            "10/10 [==============================] - 5s 472ms/step - loss: 0.6382\n",
            "Epoch 6/15\n",
            "10/10 [==============================] - 4s 432ms/step - loss: 0.6372\n",
            "Epoch 7/15\n",
            "10/10 [==============================] - 5s 478ms/step - loss: 0.6371\n",
            "Epoch 8/15\n",
            "10/10 [==============================] - 4s 435ms/step - loss: 0.6371\n",
            "Epoch 9/15\n",
            "10/10 [==============================] - 4s 434ms/step - loss: 0.6369\n",
            "Epoch 10/15\n",
            "10/10 [==============================] - 4s 423ms/step - loss: 0.6373\n",
            "Epoch 11/15\n",
            "10/10 [==============================] - 4s 426ms/step - loss: 0.6374\n",
            "Epoch 12/15\n",
            "10/10 [==============================] - 5s 476ms/step - loss: 0.6370\n",
            "Epoch 13/15\n",
            "10/10 [==============================] - 4s 428ms/step - loss: 0.6368\n",
            "Epoch 14/15\n",
            "10/10 [==============================] - 5s 468ms/step - loss: 0.6368\n",
            "Epoch 15/15\n",
            "10/10 [==============================] - 4s 433ms/step - loss: 0.6367\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QFC6ukZ9cTbp"
      },
      "source": [
        "\n",
        "# Other Resources\n",
        "\n",
        "* [Imagenet:Large Scale Visual Recognition Challenge 2014](http://image-net.org/challenges/LSVRC/2014/index)\n",
        "* [Andrej Karpathy](http://cs.stanford.edu/people/karpathy/) - PhD student/instructor at Stanford.\n",
        "    * [CS231n Convolutional Neural Networks for Visual Recognition](http://cs231n.stanford.edu/) - Stanford course on computer vision/CNN's.\n",
        "        * [CS231n - GitHub](http://cs231n.github.io/)\n",
        "    * [ConvNetJS](http://cs.stanford.edu/people/karpathy/convnetjs/) - JavaScript library for deep learning."
      ]
    }
  ]
}