kehoe_chapter09_part03_interpreting_what_convnets_learn_i.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/matthewshawnkehoe/deep-learning-with-python-notebooks/blob/master/kehoe_chapter09_part03_interpreting_what_convnets_learn_i.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ZmdjvFyKTob"
      },
      "source": [
        "This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it contains runnable code blocks and section titles, and a bunch of edits made by Matthew Kehoe.\n",
        "\n",
        "**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n",
        "\n",
        "This notebook was generated for TensorFlow 2.6."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eAY0a1SkKToc"
      },
      "source": [
        "## Problems with Computer Vision Applications\n",
        "\n",
        "- A fundamental problem when building a computer vision application\n",
        "is that of **interpretability**: why did your classifier think a particular\n",
        "image contained a bike, when all you can see is a sedan?\n",
        "- This is especially relevant to use cases where deep learning is used to\n",
        "   **complement human expertise**, such as in **medical imaging** use cases.\n",
        "- It’s often said that deep learning models are **“black boxes”**: they learn\n",
        "representations that are difficult to extract and present in a human-\n",
        "readable form.\n",
        "- Although this is partially true for certain types of deep learning\n",
        "models, it’s definitely not true for convnets."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Section 9.4 Summary\n",
        "\n",
        "The representations learned by convnets are highly amenable to\n",
        "**visualization**, in large part because they’re representations of visual\n",
        "concepts. Techniques for conceptualizing these representations include\n",
        "\n",
        "1.   **Visualizing intermediate convnet outputs** (intermediate activations)\n",
        "— Useful for understanding how successive convnet layers **transform**\n",
        "their **input**, and for getting a **first idea** of individual convnet filters.\n",
        "2.   **Visualizing convnet filters** — Useful for understanding precisely what\n",
        "visual **pattern or concept** each filter in a convnet is receptive to.\n",
        "3. **Visualizing heatmaps of class activation in an image** — Useful for\n",
        "understanding which **parts** of an image were identified as **belonging to**\n",
        "a **given class**, thus allowing you to localize objects in images.\n"
      ],
      "metadata": {
        "id": "uVUtyiDUKwiO"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qYJjk-TFKTod"
      },
      "source": [
        "### Section 4.1: Visualizing intermediate activations\n",
        "We will start by loading the Cats-versus-Dogs classification model that you saved in section 8.2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uG9JlAHKKTod",
        "outputId": "8104e559-1fdf-4775-80c6-45e6367f15b3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 39
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-d3f4b983-f974-4ff4-b1b9-7c2918155847\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-d3f4b983-f974-4ff4-b1b9-7c2918155847\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "# You can use this to load the file \"convnet_from_scratch_with_augmentation.keras\"\n",
        "# you obtained in the last chapter.\n",
        "from google.colab import files\n",
        "files.upload()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Chbh4gLhKToe"
      },
      "outputs": [],
      "source": [
        "from tensorflow import keras\n",
        "model = keras.models.load_model(\"convnet_from_scratch_with_augmentation.keras\")\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we’ll get an input image—a picture of a cat, not part of the images the network was trained on."
      ],
      "metadata": {
        "id": "C_BBCjz_ShaX"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELmWxKljKTof"
      },
      "source": [
        "**Preprocessing a single image**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZYYTMBquKTof"
      },
      "outputs": [],
      "source": [
        "from tensorflow import keras\n",
        "import numpy as np\n",
        "\n",
        "img_path = keras.utils.get_file(\n",
        "    fname=\"cat.jpg\",\n",
        "    origin=\"https://img-datasets.s3.amazonaws.com/cat.jpg\")\n",
        "\n",
        "def get_img_array(img_path, target_size):\n",
        "    img = keras.utils.load_img(\n",
        "        img_path, target_size=target_size)\n",
        "    array = keras.utils.img_to_array(img)\n",
        "    array = np.expand_dims(array, axis=0)\n",
        "    return array\n",
        "\n",
        "img_tensor = get_img_array(img_path, target_size=(180, 180))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GcpXdBGvKTof"
      },
      "source": [
        "**Displaying the test picture**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jt86VE1rKTog"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.axis(\"off\")\n",
        "plt.imshow(img_tensor[0].astype(\"uint8\"))\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gfM_hJGLKToh"
      },
      "source": [
        "**Instantiating a model that returns layer activations**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4RRKyVNwKToi"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras import layers\n",
        "\n",
        "layer_outputs = []\n",
        "layer_names = []\n",
        "for layer in model.layers:\n",
        "    if isinstance(layer, (layers.Conv2D, layers.MaxPooling2D)):\n",
        "        layer_outputs.append(layer.output)\n",
        "        layer_names.append(layer.name)\n",
        "activation_model = keras.Model(inputs=model.input, outputs=layer_outputs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fz8Rei9rKToi"
      },
      "source": [
        "**Using the model to compute layer activations**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9Vn_xronKToj"
      },
      "outputs": [],
      "source": [
        "activations = activation_model.predict(img_tensor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dU1VV_CzKToj"
      },
      "outputs": [],
      "source": [
        "first_layer_activation = activations[0]\n",
        "print(first_layer_activation.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vd_y3sCKKToj"
      },
      "source": [
        "**Visualizing the fifth channel**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qgpR3VxkKTok"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.matshow(first_layer_activation[0, :, :, 5], cmap=\"viridis\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2zbfvnKAKTok"
      },
      "source": [
        "**Visualizing every channel in every intermediate activation**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "svPQs9KBKTok"
      },
      "outputs": [],
      "source": [
        "images_per_row = 16\n",
        "for layer_name, layer_activation in zip(layer_names, activations):\n",
        "    n_features = layer_activation.shape[-1]\n",
        "    size = layer_activation.shape[1]\n",
        "    n_cols = n_features // images_per_row\n",
        "    display_grid = np.zeros(((size + 1) * n_cols - 1,\n",
        "                             images_per_row * (size + 1) - 1))\n",
        "    for col in range(n_cols):\n",
        "        for row in range(images_per_row):\n",
        "            channel_index = col * images_per_row + row\n",
        "            channel_image = layer_activation[0, :, :, channel_index].copy()\n",
        "            if channel_image.sum() != 0:\n",
        "                channel_image -= channel_image.mean()\n",
        "                channel_image /= channel_image.std()\n",
        "                channel_image *= 64\n",
        "                channel_image += 128\n",
        "            channel_image = np.clip(channel_image, 0, 255).astype(\"uint8\")\n",
        "            display_grid[\n",
        "                col * (size + 1): (col + 1) * size + col,\n",
        "                row * (size + 1) : (row + 1) * size + row] = channel_image\n",
        "    scale = 1. / size\n",
        "    plt.figure(figsize=(scale * display_grid.shape[1],\n",
        "                        scale * display_grid.shape[0]))\n",
        "    plt.title(layer_name)\n",
        "    plt.grid(False)\n",
        "    plt.axis(\"off\")\n",
        "    plt.imshow(display_grid, aspect=\"auto\", cmap=\"viridis\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JWztDoZHKTol"
      },
      "source": [
        "### Section 4.2: Visualizing convnet filters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DHJOxFhsKTol"
      },
      "source": [
        "**Instantiating the Xception convolutional base**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F927H70rKTol"
      },
      "outputs": [],
      "source": [
        "model = keras.applications.xception.Xception(\n",
        "    weights=\"imagenet\",\n",
        "    include_top=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "isVHwUtGKTol"
      },
      "source": [
        "**Printing the names of all convolutional layers in Xception**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YZ6LK7PBKTol"
      },
      "outputs": [],
      "source": [
        "for layer in model.layers:\n",
        "    if isinstance(layer, (keras.layers.Conv2D, keras.layers.SeparableConv2D)):\n",
        "        print(layer.name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R36VFa4cKTom"
      },
      "source": [
        "**Creating a feature extractor model**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1rqh9T3NKTom"
      },
      "outputs": [],
      "source": [
        "layer_name = \"block3_sepconv1\"\n",
        "layer = model.get_layer(name=layer_name)\n",
        "feature_extractor = keras.Model(inputs=model.input, outputs=layer.output)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5UFLjeOOKTon"
      },
      "source": [
        "**Using the feature extractor**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GUdy_WEXKTon"
      },
      "outputs": [],
      "source": [
        "activation = feature_extractor(\n",
        "    keras.applications.xception.preprocess_input(img_tensor)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xq1w69EOKTon"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "def compute_loss(image, filter_index):\n",
        "    activation = feature_extractor(image)\n",
        "    filter_activation = activation[:, 2:-2, 2:-2, filter_index]\n",
        "    return tf.reduce_mean(filter_activation)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DXTGR_g0KTon"
      },
      "source": [
        "**Loss maximization via stochastic gradient ascent**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "osEY-iiXKTon"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def gradient_ascent_step(image, filter_index, learning_rate):\n",
        "    with tf.GradientTape() as tape:\n",
        "        tape.watch(image)\n",
        "        loss = compute_loss(image, filter_index)\n",
        "    grads = tape.gradient(loss, image)\n",
        "    grads = tf.math.l2_normalize(grads)\n",
        "    image += learning_rate * grads\n",
        "    return image"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oDq3zBobKToo"
      },
      "source": [
        "**Function to generate filter visualizations**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6hkPsOqrKToo"
      },
      "outputs": [],
      "source": [
        "img_width = 200\n",
        "img_height = 200\n",
        "\n",
        "def generate_filter_pattern(filter_index):\n",
        "    iterations = 30\n",
        "    learning_rate = 10.\n",
        "    image = tf.random.uniform(\n",
        "        minval=0.4,\n",
        "        maxval=0.6,\n",
        "        shape=(1, img_width, img_height, 3))\n",
        "    for i in range(iterations):\n",
        "        image = gradient_ascent_step(image, filter_index, learning_rate)\n",
        "    return image[0].numpy()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YJtSPILAKToo"
      },
      "source": [
        "**Utility function to convert a tensor into a valid image**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0K3nirVRKToo"
      },
      "outputs": [],
      "source": [
        "def deprocess_image(image):\n",
        "    image -= image.mean()\n",
        "    image /= image.std()\n",
        "    image *= 64\n",
        "    image += 128\n",
        "    image = np.clip(image, 0, 255).astype(\"uint8\")\n",
        "    image = image[25:-25, 25:-25, :]\n",
        "    return image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "meM6hspCKTop"
      },
      "outputs": [],
      "source": [
        "plt.axis(\"off\")\n",
        "plt.imshow(deprocess_image(generate_filter_pattern(filter_index=2)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Kd2H2tBKTop"
      },
      "source": [
        "**Generating a grid of all filter response patterns in a layer**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QVIWaqmlKTop"
      },
      "outputs": [],
      "source": [
        "all_images = []\n",
        "for filter_index in range(64):\n",
        "    print(f\"Processing filter {filter_index}\")\n",
        "    image = deprocess_image(\n",
        "        generate_filter_pattern(filter_index)\n",
        "    )\n",
        "    all_images.append(image)\n",
        "\n",
        "margin = 5\n",
        "n = 8\n",
        "cropped_width = img_width - 25 * 2\n",
        "cropped_height = img_height - 25 * 2\n",
        "width = n * cropped_width + (n - 1) * margin\n",
        "height = n * cropped_height + (n - 1) * margin\n",
        "stitched_filters = np.zeros((width, height, 3))\n",
        "\n",
        "for i in range(n):\n",
        "    for j in range(n):\n",
        "        image = all_images[i * n + j]\n",
        "        stitched_filters[\n",
        "            (cropped_width + margin) * i : (cropped_width + margin) * i + cropped_width,\n",
        "            (cropped_height + margin) * j : (cropped_height + margin) * j\n",
        "            + cropped_height,\n",
        "            :,\n",
        "        ] = image\n",
        "\n",
        "keras.utils.save_img(\n",
        "    f\"filters_for_layer_{layer_name}.png\", stitched_filters)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-bxyl2M_KTop"
      },
      "source": [
        "### Section 4.3: Visualizing heatmaps of class activation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fBs1c4m2KToq"
      },
      "source": [
        "**Loading the Xception network with pretrained weights**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-Ln9XQQaKToq"
      },
      "outputs": [],
      "source": [
        "model = keras.applications.xception.Xception(weights=\"imagenet\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5u8ey0G3KToq"
      },
      "source": [
        "**Preprocessing an input image for Xception**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vYcEP9wDKToq"
      },
      "outputs": [],
      "source": [
        "img_path = keras.utils.get_file(\n",
        "    fname=\"elephant.jpg\",\n",
        "    origin=\"https://img-datasets.s3.amazonaws.com/elephant.jpg\")\n",
        "\n",
        "def get_img_array(img_path, target_size):\n",
        "    img = keras.utils.load_img(img_path, target_size=target_size)\n",
        "    array = keras.utils.img_to_array(img)\n",
        "    array = np.expand_dims(array, axis=0)\n",
        "    array = keras.applications.xception.preprocess_input(array)\n",
        "    return array\n",
        "\n",
        "img_array = get_img_array(img_path, target_size=(299, 299))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-J13Jm4tKTor"
      },
      "outputs": [],
      "source": [
        "preds = model.predict(img_array)\n",
        "print(keras.applications.xception.decode_predictions(preds, top=3)[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PELXa0rdKTor"
      },
      "outputs": [],
      "source": [
        "np.argmax(preds[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hfxBfsLhKTor"
      },
      "source": [
        "**Setting up a model that returns the last convolutional output**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IthHuwilKTor"
      },
      "outputs": [],
      "source": [
        "last_conv_layer_name = \"block14_sepconv2_act\"\n",
        "classifier_layer_names = [\n",
        "    \"avg_pool\",\n",
        "    \"predictions\",\n",
        "]\n",
        "last_conv_layer = model.get_layer(last_conv_layer_name)\n",
        "last_conv_layer_model = keras.Model(model.inputs, last_conv_layer.output)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y-bKuBX0KTos"
      },
      "source": [
        "**Reapplying the classifier on top of the last convolutional output**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0geNVJbkKTos"
      },
      "outputs": [],
      "source": [
        "classifier_input = keras.Input(shape=last_conv_layer.output.shape[1:])\n",
        "x = classifier_input\n",
        "for layer_name in classifier_layer_names:\n",
        "    x = model.get_layer(layer_name)(x)\n",
        "classifier_model = keras.Model(classifier_input, x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "95DR214pKTos"
      },
      "source": [
        "**Retrieving the gradients of the top predicted class**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xQqeQhkpKTos"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "with tf.GradientTape() as tape:\n",
        "    last_conv_layer_output = last_conv_layer_model(img_array)\n",
        "    tape.watch(last_conv_layer_output)\n",
        "    preds = classifier_model(last_conv_layer_output)\n",
        "    top_pred_index = tf.argmax(preds[0])\n",
        "    top_class_channel = preds[:, top_pred_index]\n",
        "\n",
        "grads = tape.gradient(top_class_channel, last_conv_layer_output)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ng0eetGCKTot"
      },
      "source": [
        "**Gradient pooling and channel-importance weighting**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TguuyNPTKTot"
      },
      "outputs": [],
      "source": [
        "pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2)).numpy()\n",
        "last_conv_layer_output = last_conv_layer_output.numpy()[0]\n",
        "for i in range(pooled_grads.shape[-1]):\n",
        "    last_conv_layer_output[:, :, i] *= pooled_grads[i]\n",
        "heatmap = np.mean(last_conv_layer_output, axis=-1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "95fRj6SaKTot"
      },
      "source": [
        "**Heatmap post-processing**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GfFLEip2KTot"
      },
      "outputs": [],
      "source": [
        "heatmap = np.maximum(heatmap, 0)\n",
        "heatmap /= np.max(heatmap)\n",
        "plt.matshow(heatmap)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sbf1Y-ZxKTot"
      },
      "source": [
        "**Superimposing the heatmap on the original picture**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e2aKg3l5KTou"
      },
      "outputs": [],
      "source": [
        "import matplotlib.cm as cm\n",
        "\n",
        "img = keras.utils.load_img(img_path)\n",
        "img = keras.utils.img_to_array(img)\n",
        "\n",
        "heatmap = np.uint8(255 * heatmap)\n",
        "\n",
        "jet = cm.get_cmap(\"jet\")\n",
        "jet_colors = jet(np.arange(256))[:, :3]\n",
        "jet_heatmap = jet_colors[heatmap]\n",
        "\n",
        "jet_heatmap = keras.utils.array_to_img(jet_heatmap)\n",
        "jet_heatmap = jet_heatmap.resize((img.shape[1], img.shape[0]))\n",
        "jet_heatmap = keras.utils.img_to_array(jet_heatmap)\n",
        "\n",
        "superimposed_img = jet_heatmap * 0.4 + img\n",
        "superimposed_img = keras.utils.array_to_img(superimposed_img)\n",
        "\n",
        "save_path = \"elephant_cam.jpg\"\n",
        "superimposed_img.save(save_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5R4RKs_iKTou"
      },
      "source": [
        "## Summary"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}