LSTM_LSTM_training.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "### CS Interview QA Chatbot Usage\n",
        "This workbook contains the code for training and running the a LSTM-RNN model that answers question about CS interview questions.\n",
        "\n",
        "To run the saved model:\n",
        "1. [Import libraries](#scrollTo=aO1Gqpuvtmud&line=1&uniqifier=1)\n",
        "1. [Load the preprocessed dataset](#scrollTo=G3ow6JgothNy)\n",
        "1. [Run the utility functions](#scrollTo=-NASuplC3w-N&line=4&uniqifier=1)\n",
        "1. [Define the model architecture](#scrollTo=JTZIHoO-41rf&line=11&uniqifier=1)\n",
        "1. [Load saved model](#scrollTo=3HQZcsVhKU8I&line=1&uniqifier=1)\n",
        "1. [Run evaluation functions](#scrollTo=03W05HyA5Sxh&line=1&uniqifier=1)\n",
        "1. [Evaluate manually (qualitative)](#scrollTo=H_GOPvu_YYwr)\n",
        "    \n",
        "    Sample questions from the dataset\n",
        "        1. what is recursion\n",
        "        2. how do you explain bestfirst search\n",
        "        3. how do you explain a thread\n",
        "        4. how does divide and conquer work\n",
        "1. [Evaluate with BERT Score (quantitative)](#scrollTo=n2WXvkcxkj6z)\n",
        "\n",
        "\n",
        "To train from scratch:\n",
        "1. [Import libraries](#scrollTo=aO1Gqpuvtmud&line=1&uniqifier=1)\n",
        "1. [Load the preprocessed dataset](#scrollTo=G3ow6JgothNy)\n",
        "1. [Run the utility functions](#scrollTo=-NASuplC3w-N&line=4&uniqifier=1)\n",
        "1. [Define the model architecture](#scrollTo=JTZIHoO-41rf&line=11&uniqifier=1)\n",
        "1. [Run the training functions](#scrollTo=SFjeoyzLfRmz)\n",
        "1. [Initialise model](#scrollTo=3HQZcsVhKU8I&line=1&uniqifier=1)    \n",
        "    Set the `loadFilename` variable to `None`\n",
        "1. [Run evaluation functions](#scrollTo=03W05HyA5Sxh&line=1&uniqifier=1)\n",
        "1. [Evaluate manually (qualitative)](#scrollTo=H_GOPvu_YYwr)\n",
        "1. [Evaluate with BERT Score (quantitative)](#scrollTo=n2WXvkcxkj6z)"
      ],
      "metadata": {
        "id": "_M89VKPPqpTo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Imports"
      ],
      "metadata": {
        "id": "aO1Gqpuvtmud"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive', force_remount=True)\n",
        "FOLDER_PATH = '/content/drive/My Drive/Colab Notebooks/nlc/project'"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oB5MazNu379Q",
        "outputId": "43dcb17f-a4a9-4060-ddb8-0b3a7d65d80b"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "from __future__ import unicode_literals\n",
        "\n",
        "import torch\n",
        "from torch.jit import script, trace\n",
        "import torch.nn as nn\n",
        "from torch import optim\n",
        "import torch.nn.functional as F\n",
        "import csv\n",
        "import random\n",
        "import re\n",
        "import os\n",
        "import unicodedata\n",
        "import codecs\n",
        "from io import open\n",
        "import itertools\n",
        "import math\n",
        "import pandas as pd\n",
        "\n",
        "\n",
        "USE_CUDA = torch.cuda.is_available()\n",
        "device = torch.device(\"cuda\" if USE_CUDA else \"cpu\")"
      ],
      "metadata": {
        "id": "pHkIyAN5U4j0"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Loading Saved Data"
      ],
      "metadata": {
        "id": "G3ow6JgothNy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "# read from csv\n",
        "df = pd.read_csv(os.path.join(FOLDER_PATH, 'combined_data.csv'))\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "VWYGjdz_tjVW",
        "outputId": "585fbf96-19e0-4d30-bcb0-97fcf37c1196"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                               Question  \\\n",
              "0                    how does randomised algorithm work   \n",
              "1                  what do you mean by bestfirst search   \n",
              "2                           how do you explain a daemon   \n",
              "3                            what is phonetic algorithm   \n",
              "4                what do you mean by uniform costsearch   \n",
              "...                                                 ...   \n",
              "3770                      explain biasvariance tradeoff   \n",
              "3771  what is stochastic gradient descent sgd in mac...   \n",
              "3772                explain stochastic gradient descent   \n",
              "3773  what is the backpropagation algorithm in machi...   \n",
              "3774                  explain backpropagation algorithm   \n",
              "\n",
              "                                                 Answer  \n",
              "0     the algorithm typically uses uniformly random ...  \n",
              "1     bestfirst search is a search algorithm which e...  \n",
              "2     daemon disk and execution monitor is a process...  \n",
              "3     a phonetic algorithm is an algorithm for index...  \n",
              "4     a tree search that finds the lowestcost route ...  \n",
              "...                                                 ...  \n",
              "3770  biasvariance tradeoff is a concept in machine ...  \n",
              "3771  stochastic gradient descent sgd is an optimiza...  \n",
              "3772  stochastic gradient descent sgd is an optimiza...  \n",
              "3773  the backpropagation algorithm is a widely used...  \n",
              "3774  the backpropagation algorithm is a widely used...  \n",
              "\n",
              "[3775 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ac30c65a-7993-4428-bcb3-a19a15b511cb\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Question</th>\n",
              "      <th>Answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>how does randomised algorithm work</td>\n",
              "      <td>the algorithm typically uses uniformly random ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>what do you mean by bestfirst search</td>\n",
              "      <td>bestfirst search is a search algorithm which e...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>how do you explain a daemon</td>\n",
              "      <td>daemon disk and execution monitor is a process...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>what is phonetic algorithm</td>\n",
              "      <td>a phonetic algorithm is an algorithm for index...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>what do you mean by uniform costsearch</td>\n",
              "      <td>a tree search that finds the lowestcost route ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3770</th>\n",
              "      <td>explain biasvariance tradeoff</td>\n",
              "      <td>biasvariance tradeoff is a concept in machine ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3771</th>\n",
              "      <td>what is stochastic gradient descent sgd in mac...</td>\n",
              "      <td>stochastic gradient descent sgd is an optimiza...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3772</th>\n",
              "      <td>explain stochastic gradient descent</td>\n",
              "      <td>stochastic gradient descent sgd is an optimiza...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3773</th>\n",
              "      <td>what is the backpropagation algorithm in machi...</td>\n",
              "      <td>the backpropagation algorithm is a widely used...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3774</th>\n",
              "      <td>explain backpropagation algorithm</td>\n",
              "      <td>the backpropagation algorithm is a widely used...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3775 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ac30c65a-7993-4428-bcb3-a19a15b511cb')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ac30c65a-7993-4428-bcb3-a19a15b511cb button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ac30c65a-7993-4428-bcb3-a19a15b511cb');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-b46e67a5-1dfc-4923-82c8-a92b3b264df1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b46e67a5-1dfc-4923-82c8-a92b3b264df1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-b46e67a5-1dfc-4923-82c8-a92b3b264df1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_983a0cc6-4dcc-4c5c-ab2e-578c94089120\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_983a0cc6-4dcc-4c5c-ab2e-578c94089120 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 3775,\n  \"fields\": [\n    {\n      \"column\": \"Question\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3299,\n        \"samples\": [\n          \"define bidirectional search\",\n          \"what do you mean by divide and conquer algorithm\",\n          \"explain accumulatorbased architecture in computer systems\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Answer\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1817,\n        \"samples\": [\n          \"inner join returns matching rows from both tables left join returns all rows from the left table and matching rows from the right\",\n          \"the basic operations carried out in a linked list include creation of a listinsertion of a nodedeletion of a nodemodification of a nodetraversal of the list\",\n          \"there is no concept of folder in ubuntu everything included in your hardware is a file\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# convert the df to 2d list\n",
        "qa_pairs = df.values.tolist()\n",
        "len(qa_pairs), qa_pairs[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tsHfppjCpu-a",
        "outputId": "51655a35-863d-49dd-b44d-774d170ccbdb"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3775,\n",
              " ['how does randomised algorithm work',\n",
              "  'the algorithm typically uses uniformly random bits as an auxiliary input to guide its behavior in the hope of achieving good performance in the average case over all possible choices of random bits'])"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Utility Functions\n",
        "- Create vocabulary\n",
        "- Normalising string\n",
        "- Prepare data in batches"
      ],
      "metadata": {
        "id": "-NASuplC3w-N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Default word tokens\n",
        "PAD_token = 0  # Used for padding short sentences\n",
        "SOS_token = 1  # Start-of-sentence token\n",
        "EOS_token = 2  # End-of-sentence token\n",
        "UNK_token = 3  # Unknown word token\n",
        "\n",
        "class Voc:\n",
        "    def __init__(self, name):\n",
        "        self.name = name\n",
        "        self.trimmed = False\n",
        "        self.word2index = {}\n",
        "        self.word2count = {}\n",
        "        self.index2word = {PAD_token: \"PAD\", SOS_token: \"SOS\", EOS_token: \"EOS\", UNK_token: \"UNK\"}\n",
        "        self.num_words = 4  # Count SOS, EOS, PAD, UNK\n",
        "\n",
        "    def addSentence(self, sentence):\n",
        "        for word in sentence.split(' '):\n",
        "            self.addWord(word)\n",
        "\n",
        "    def addWord(self, word):\n",
        "        if word not in self.word2index:\n",
        "            self.word2index[word] = self.num_words\n",
        "            self.word2count[word] = 1\n",
        "            self.index2word[self.num_words] = word\n",
        "            self.num_words += 1\n",
        "        else:\n",
        "            self.word2count[word] += 1\n",
        "\n",
        "    # Remove words below a certain count threshold\n",
        "    def trim(self, min_count):\n",
        "        if self.trimmed:\n",
        "            return\n",
        "        self.trimmed = True\n",
        "\n",
        "        keep_words = []\n",
        "\n",
        "        for k, v in self.word2count.items():\n",
        "            if v >= min_count:\n",
        "                keep_words.append(k)\n",
        "\n",
        "        print('keep_words {} / {} = {:.4f}'.format(\n",
        "            len(keep_words), len(self.word2index), len(keep_words) / len(self.word2index)\n",
        "        ))\n",
        "\n",
        "        # Reinitialize dictionaries\n",
        "        self.word2index = {}\n",
        "        self.word2count = {}\n",
        "        self.index2word = {PAD_token: \"PAD\", SOS_token: \"SOS\", EOS_token: \"EOS\", UNK_token: \"UNK\"}\n",
        "        self.num_words = 4 # Count default tokens\n",
        "\n",
        "        for word in keep_words:\n",
        "            self.addWord(word)"
      ],
      "metadata": {
        "id": "pY5GHdd0Tovm"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Turn a Unicode string to plain ASCII, thanks to\n",
        "# http://stackoverflow.com/a/518232/2809427\n",
        "def unicodeToAscii(s):\n",
        "    return ''.join(\n",
        "        c for c in unicodedata.normalize('NFD', s)\n",
        "        if unicodedata.category(c) != 'Mn'\n",
        "    )\n",
        "\n",
        "# Lowercase, trim, and remove non-letter characters\n",
        "def normalizeString(s):\n",
        "    s = unicodeToAscii(s.lower().strip())\n",
        "    s = re.sub(r\"([.!?])\", r\" \\1\", s)\n",
        "    s = re.sub(r\"[^a-zA-Z.!?]+\", r\" \", s)\n",
        "    s = re.sub(r\"\\s+\", r\" \", s).strip()\n",
        "    return s\n",
        "\n",
        "# Returns True iff both sentences in a pair 'p' are under the MAX_LENGTH threshold\n",
        "def filterPair(p, max_length):\n",
        "    # Input sequences need to preserve the last word for EOS token\n",
        "    return len(p[0].split(' ')) < max_length and len(p[1].split(' ')) < max_length\n",
        "\n",
        "# Filter pairs using filterPair condition\n",
        "def filterPairs(pairs, max_length):\n",
        "    return [pair for pair in pairs if filterPair(pair, max_length)]\n"
      ],
      "metadata": {
        "id": "SEwUmgIBUn_1"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "voc = Voc(\"qa\")\n",
        "MAX_LENGTH = 256  # Maximum sentence length to consider\n",
        "\n",
        "# filter pairs based on MAX_LENGTH\n",
        "print(f'before filtering, no. of qa pairs: {len(qa_pairs)}')\n",
        "qa_pairs = filterPairs(qa_pairs, MAX_LENGTH)\n",
        "print(f'after filtering, no. of qa pairs: {len(qa_pairs)}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nVcfYDvrqYDg",
        "outputId": "b06ae399-34ab-40e9-c4e2-24bca664bd10"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "before filtering, no. of qa pairs: 3775\n",
            "after filtering, no. of qa pairs: 3775\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# normalise each qa pair and add word from question and answer into vocab\n",
        "for i in range(len(qa_pairs)):\n",
        "    question, answer = qa_pairs[i]\n",
        "    question = normalizeString(question)\n",
        "    answer = normalizeString(answer)\n",
        "    voc.addSentence(question)\n",
        "    voc.addSentence(answer)\n",
        "    qa_pairs[i] = [question, answer]\n",
        "\n",
        "vocab_size = voc.num_words # words + 3 tokens (PAD, SOS, EOS)\n",
        "questions = [pair[0] for pair in qa_pairs]\n",
        "answers = [pair[1] for pair in qa_pairs]\n",
        "\n",
        "# preview first 5 questions and answers\n",
        "qa_pairs[:5]"
      ],
      "metadata": {
        "id": "zbcg0ggv41Hs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fd382db5-ea0b-4172-d903-820e38a04744"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[['how does randomised algorithm work',\n",
              "  'the algorithm typically uses uniformly random bits as an auxiliary input to guide its behavior in the hope of achieving good performance in the average case over all possible choices of random bits'],\n",
              " ['what do you mean by bestfirst search',\n",
              "  'bestfirst search is a search algorithm which explores a graph by expanding the most promising node chosen according to a specified rule'],\n",
              " ['how do you explain a daemon',\n",
              "  'daemon disk and execution monitor is a process that runs in the background without users interaction they usually start at the booting time and terminate when the system is shut down'],\n",
              " ['what is phonetic algorithm',\n",
              "  'a phonetic algorithm is an algorithm for indexing of words by their pronunciation'],\n",
              " ['what do you mean by uniform costsearch',\n",
              "  'a tree search that finds the lowestcost route where costs vary']]"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import random\n",
        "import itertools\n",
        "\n",
        "def indexesFromSentence(voc, sentence):\n",
        "    \"\"\"Convert a sentence into a list of word indices, handling unknown words.\"\"\"\n",
        "    return [voc.word2index.get(word, UNK_token) for word in sentence.split(' ')] + [EOS_token]\n",
        "\n",
        "def zeroPadding(l, fillvalue=PAD_token):\n",
        "    return list(itertools.zip_longest(*l, fillvalue=fillvalue))\n",
        "\n",
        "def binaryMatrix(l, value=PAD_token):\n",
        "    m = []\n",
        "    for i, seq in enumerate(l):\n",
        "        m.append([])\n",
        "        for token in seq:\n",
        "            if token == PAD_token:\n",
        "                m[i].append(0)\n",
        "            else:\n",
        "                m[i].append(1)\n",
        "    return m\n",
        "\n",
        "# Returns padded input sequence tensor and lengths\n",
        "def inputVar(l, voc):\n",
        "    indexes_batch = [indexesFromSentence(voc, sentence) for sentence in l]\n",
        "    # print('indexes_batch', indexes_batch) # indexes_batch [[34, 35, 36, 37, 38, 39, 40, 2], [3, 4, 5, 6, 7, 2]]\n",
        "    lengths = torch.tensor([len(indexes) for indexes in indexes_batch])\n",
        "    padList = zeroPadding(indexes_batch)\n",
        "    # print('inputpadList', padList) # padList [(34, 3), (35, 4), (36, 5), (37, 6), (38, 7), (39, 2), (40, 0), (2, 0)]\n",
        "    padVar = torch.LongTensor(padList)\n",
        "    return padVar, lengths\n",
        "\n",
        "# Returns padded target sequence tensor, padding mask, and max target length\n",
        "def outputVar(l, voc):\n",
        "    indexes_batch = [indexesFromSentence(voc, sentence) for sentence in l]\n",
        "    max_target_len = max([len(indexes) for indexes in indexes_batch])\n",
        "    padList = zeroPadding(indexes_batch)\n",
        "    # print('outputpadList',padList) # padList [(1669, 949), (35, 36), (26, 2), (2, 0)]\n",
        "    mask = binaryMatrix(padList) # 1 for non-pad tokens, 0 for pad tokens\n",
        "    # print('mask', mask) # mask [[1, 1], [1, 1], [1, 1], [1, 0]]\n",
        "    mask = torch.ByteTensor(mask)\n",
        "    # print('mask', mask) # mask [[1, 1], [1, 1], [1, 1], [1, 0]]\n",
        "    padVar = torch.LongTensor(padList)\n",
        "    return padVar, mask, max_target_len\n",
        "\n",
        "# Returns all items for a given batch of pairs\n",
        "def batch2TrainData(voc, pair_batch):\n",
        "    pair_batch.sort(key=lambda x: len(x[0].split(\" \")), reverse=True)\n",
        "    input_batch, output_batch = [], []\n",
        "    for pair in pair_batch:\n",
        "        input_batch.append(pair[0])\n",
        "        output_batch.append(pair[1])\n",
        "    # print('input_batch', input_batch) # input_batch ['what do you mean by bestfirst search', 'how does randomised algorithm work']\n",
        "    inp, lengths = inputVar(input_batch, voc)\n",
        "    output, mask, max_target_len = outputVar(output_batch, voc)\n",
        "    return inp, lengths, output, mask, max_target_len\n",
        "\n",
        "\n",
        "# # Example for validation\n",
        "# testing_pairs = [['how are you doing', 'i do good'], ['you are good', 'like you']]\n",
        "# small_batch_size = 2\n",
        "# # batches = batch2TrainData(voc, [random.choice(qa_pairs) for _ in range(small_batch_size)])\n",
        "# # batches = batch2TrainData(voc, qa_pairs[:small_batch_size])\n",
        "# batches = batch2TrainData(voc, testing_pairs)\n",
        "# input_variable, lengths, target_variable, mask, max_target_len = batches\n",
        "\n",
        "# print(\"input_variable:\", input_variable) # shape (max_input_sequence_length, batch_size)\n",
        "# print(\"lengths:\", lengths)\n",
        "# print(\"target_variable:\", target_variable) # shape (max_target_len, batch_size)\n",
        "# print(\"mask:\", mask)\n",
        "# print(\"max_target_len:\", max_target_len)"
      ],
      "metadata": {
        "id": "31J1CNupTmxt"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Defining model architecture\n",
        "\n",
        "**Computation Process:**\n",
        "    \n",
        "    1) Convert word indexes to embeddings.\n",
        "    2) Pack padded batch of sequences for RNN module.\n",
        "    3) Forward pass through LSTM.\n",
        "    4) Unpack padding.\n",
        "    5) Sum bidirectional LSTM outputs.\n",
        "    6) Return output and final hidden state.\n"
      ],
      "metadata": {
        "id": "JTZIHoO-41rf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Encoder"
      ],
      "metadata": {
        "id": "Xk3OsS0uau5P"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class EncoderRNN(nn.Module):\n",
        "    def __init__(self, hidden_size, embedding, n_layers=1, dropout=0):\n",
        "        super(EncoderRNN, self).__init__()\n",
        "        self.n_layers = n_layers\n",
        "        self.hidden_size = hidden_size\n",
        "        self.embedding = embedding\n",
        "\n",
        "        self.lstm = nn.LSTM(hidden_size, hidden_size, n_layers,\n",
        "                            dropout=(0 if n_layers == 1 else dropout), bidirectional=True)\n",
        "\n",
        "    def forward(self, input_seq, input_lengths, hidden=None):\n",
        "        # Convert word indexes to embeddings\n",
        "        embedded = self.embedding(input_seq) # output shape (L, N, embedding_dim); output shape (*, embedding_dim) where * is the shape of the input_seq\n",
        "        # Pack padded batch of sequences for RNN module\n",
        "        packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths)\n",
        "        # Forward pass through LSTM\n",
        "        outputs, hidden = self.lstm(packed, hidden) # input shape (L, N, hidden_size) or packed sequence object\n",
        "        # Unpack padding\n",
        "        outputs, _ = torch.nn.utils.rnn.pad_packed_sequence(outputs)\n",
        "        # Sum bidirectional LSTM outputs.\n",
        "        # Note that here the outputs from bigru will have the shape = (max_length, batch_size, hidden_size x num_directions)\n",
        "        # And we want to sum the output from different directions at the same time step.\n",
        "        outputs = outputs[:, :, :self.hidden_size] + outputs[:, : ,self.hidden_size:]\n",
        "        # Return output and final hidden state\n",
        "        return outputs, hidden # output shape is (max_length, batch_size, hidden_size) where hidden_size is sum of the bidirection outputs"
      ],
      "metadata": {
        "id": "L632EefO41vx"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Attention layer"
      ],
      "metadata": {
        "id": "Yb4wHngnaxzQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Luong attention layer\n",
        "class Attn(torch.nn.Module):\n",
        "    def __init__(self, method, hidden_size):\n",
        "        super(Attn, self).__init__()\n",
        "        self.method = method\n",
        "        self.hidden_size = hidden_size\n",
        "        if self.method == 'general':\n",
        "            self.attn = torch.nn.Linear(self.hidden_size, hidden_size)\n",
        "        elif self.method == 'concat':\n",
        "            self.attn = torch.nn.Linear(self.hidden_size * 2, hidden_size)\n",
        "            self.v = torch.nn.Parameter(torch.FloatTensor(hidden_size))\n",
        "\n",
        "    def dot_score(self, hidden, encoder_output):\n",
        "        return torch.sum(hidden * encoder_output, dim=2)\n",
        "\n",
        "    def general_score(self, hidden, encoder_output):\n",
        "        energy = self.attn(encoder_output)\n",
        "        return torch.sum(hidden * energy, dim=2)\n",
        "\n",
        "    def concat_score(self, hidden, encoder_output):\n",
        "        # hidden.expand(encoder_output.size(0), -1, -1): hidden (1, batch_size, hidden_size) -> (max_len, batch_size, hidden_size)\n",
        "        # after cat: (max_len, batch_size, 2 * hidden_size)\n",
        "        # after self.attn: (max_len, batch_size, hidden_size)\n",
        "        energy = self.attn(torch.cat((hidden.expand(encoder_output.size(0), -1, -1), encoder_output), 2)).tanh()\n",
        "        return torch.sum(self.v * energy, dim=2)\n",
        "\n",
        "    def forward(self, hidden, encoder_outputs):\n",
        "        # Calculate the attention weights (energies) based on the given method\n",
        "        # attn_energies has shape = (max_length, batch_size)\n",
        "        if self.method == 'general':\n",
        "            attn_energies = self.general_score(hidden, encoder_outputs)\n",
        "        elif self.method == 'concat':\n",
        "            attn_energies = self.concat_score(hidden, encoder_outputs)\n",
        "        elif self.method == 'dot':\n",
        "            attn_energies = self.dot_score(hidden, encoder_outputs)\n",
        "\n",
        "        # Transpose max_length and batch_size dimensions\n",
        "        attn_energies = attn_energies.t() # attn_energies shape is (max_length, batch_size), after transpose is (batch_size, max_len)\n",
        "\n",
        "        # Return the softmax normalized probability scores (with added dimension) # (zhiwei's comment: softmax across dimension 1 means finding the attention weights for each word (dimension 1 is the max_length i.e. sequence length)\n",
        "        return F.softmax(attn_energies, dim=1).unsqueeze(1) # (batch_size,1,max_length)"
      ],
      "metadata": {
        "id": "aFpXFhaOKPPR"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Decoder\n",
        "\n",
        "**Computation Process:**\n",
        "\n",
        "    1) Get embedding of current input word.\n",
        "    2) Forward through unidirectional LSTM.\n",
        "    3) Calculate attention weights from the current LSTM output from (2).\n",
        "    4) Multiply attention weights to encoder outputs to get new \"weighted sum\" context vector.\n",
        "    5) Concatenate weighted context vector and LSTM output using Luong\n",
        "    6) Predict next word using Luong eq. 6 (without softmax).\n",
        "    7) Return output and final hidden state."
      ],
      "metadata": {
        "id": "S1WJJ8FkazeA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class LuongAttnDecoderRNN(nn.Module):\n",
        "    def __init__(self, attn_model, embedding, hidden_size, output_size, n_layers=1, dropout=0.1):\n",
        "        super(LuongAttnDecoderRNN, self).__init__()\n",
        "\n",
        "        # Keep for reference\n",
        "        self.attn_model = attn_model\n",
        "        self.hidden_size = hidden_size\n",
        "        self.output_size = output_size\n",
        "        self.n_layers = n_layers\n",
        "        self.dropout = dropout\n",
        "\n",
        "        # Define layers\n",
        "        self.embedding = embedding\n",
        "        self.embedding_dropout = nn.Dropout(dropout)\n",
        "        self.lstm = nn.LSTM(hidden_size, hidden_size, n_layers, dropout=(0 if n_layers == 1 else dropout))\n",
        "        self.concat = nn.Linear(hidden_size * 2, hidden_size)\n",
        "        self.out = nn.Linear(hidden_size, output_size)\n",
        "\n",
        "        self.attn = Attn(attn_model, hidden_size)\n",
        "\n",
        "    def forward(self, input_seq, last_hidden, encoder_outputs):\n",
        "        # Note: we run this one step (word) at a time (across batch)\n",
        "        # Get embedding of current input word\n",
        "        embedded = self.embedding(input_seq)\n",
        "        embedded = self.embedding_dropout(embedded)\n",
        "        rnn_output, hidden = self.lstm(embedded, last_hidden)\n",
        "        # Calculate attention weights from the current LSTM output\n",
        "        attn_weights = self.attn(rnn_output, encoder_outputs) # attn_weights shape (batch_size,1,max_length), where max_length is the softmax attn weights for each input token\n",
        "        # Multiply attention weights to encoder outputs to get new \"weighted sum\" context vector\n",
        "        context = attn_weights.bmm(encoder_outputs.transpose(0, 1)) # (batch_size, 1, max_length) X (batch_size, max_length, hidden_size) -> (batch_size, 1, hidden_size)\n",
        "        # Concatenate weighted context vector and LSTM output using Luong eq. 5\n",
        "        rnn_output = rnn_output.squeeze(0) # (1, batch_size, hidden_size)->(batch_size, hidden_size)\n",
        "        context = context.squeeze(1) # (batch_size, hidden_size)\n",
        "        concat_input = torch.cat((rnn_output, context), 1) # (batch_size, 2 * hidden_size)\n",
        "        concat_output = torch.tanh(self.concat(concat_input)) # (batch_size, 2 * hidden_size) # tanh activation on the concatenated context and rnn output\n",
        "        # Predict next word using Luong eq. 6\n",
        "        output = self.out(concat_output) # (batch_size, hidden_size)\n",
        "        output = F.softmax(output, dim=1) # (batch_size, output_size)\n",
        "        # Return output and final hidden state\n",
        "        return output, hidden # output shape (batch_size, output_size), hidden (h_n, c_n) where h_n (D * num_layers, N, h_out), c_n (D * num_layers, N, h_cell)"
      ],
      "metadata": {
        "id": "uJgxd-N5LwBi"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Training Functions"
      ],
      "metadata": {
        "id": "SFjeoyzLfRmz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def maskNLLLoss(inp, target, mask): # inp is decoder_output which is softmax prob. of each word. shape (batch_size, voc.num_words)\n",
        "    nTotal = mask.sum()\n",
        "    crossEntropy = -torch.log(torch.gather(inp, 1, target.view(-1, 1)))\n",
        "    loss = crossEntropy.masked_select(mask).mean()\n",
        "    loss = loss.to(device)\n",
        "    return loss, nTotal.item()"
      ],
      "metadata": {
        "id": "OY_oRZk6SJ5K"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def train(input_variable, lengths, target_variable, mask, max_target_len, encoder, decoder, embedding,\n",
        "          encoder_optimizer, decoder_optimizer, batch_size, clip, max_length=MAX_LENGTH):\n",
        "\n",
        "    # Zero gradients\n",
        "    encoder_optimizer.zero_grad()\n",
        "    decoder_optimizer.zero_grad()\n",
        "\n",
        "    # Set device options\n",
        "    input_variable = input_variable.to(device)\n",
        "    # lengths = lengths.to(device)\n",
        "    target_variable = target_variable.to(device)\n",
        "    mask = mask.to(device)\n",
        "\n",
        "    # Initialize variables\n",
        "    loss = 0\n",
        "    print_losses = []\n",
        "    n_totals = 0\n",
        "\n",
        "    # Forward pass through encoder\n",
        "    encoder_outputs, encoder_hidden = encoder(input_variable, lengths)\n",
        "\n",
        "    # Create initial decoder input (start with SOS tokens for each sentence)\n",
        "    decoder_input = torch.LongTensor([[SOS_token for _ in range(batch_size)]])\n",
        "    decoder_input = decoder_input.to(device)\n",
        "\n",
        "    # Set initial decoder hidden state to the encoder's final hidden state\n",
        "    # decoder_hidden = encoder_hidden[:decoder.n_layers]\n",
        "    decoder_hidden = (encoder_hidden[0][:decoder.n_layers], encoder_hidden[1][:decoder.n_layers])\n",
        "    # print('decoder_hidden shape', decoder_hidden.shape)\n",
        "\n",
        "    # Determine if we are using teacher forcing this iteration\n",
        "    use_teacher_forcing = True if random.random() < teacher_forcing_ratio else False\n",
        "\n",
        "    # Forward batch of sequences through decoder one time step at a time\n",
        "    if use_teacher_forcing:\n",
        "        for t in range(max_target_len):\n",
        "            # print('decoder_input.shape', decoder_input.shape)\n",
        "            decoder_output, decoder_hidden = decoder(\n",
        "                decoder_input, decoder_hidden, encoder_outputs\n",
        "            )\n",
        "            # Teacher forcing: next input is current target\n",
        "            decoder_input = target_variable[t].view(1, -1) # (zw's comment) target_variable[t] means length t (next one) since index start from 0\n",
        "            # Calculate and accumulate loss\n",
        "            mask_loss, nTotal = maskNLLLoss(decoder_output, target_variable[t], mask[t])\n",
        "            loss += mask_loss\n",
        "            print_losses.append(mask_loss.item() * nTotal)\n",
        "            n_totals += nTotal\n",
        "\n",
        "    else:\n",
        "        for t in range(max_target_len):\n",
        "            decoder_output, decoder_hidden = decoder(\n",
        "                decoder_input, decoder_hidden, encoder_outputs\n",
        "            )\n",
        "            # No teacher forcing: next input is decoder's own current output\n",
        "            _, topi = decoder_output.topk(1)\n",
        "            decoder_input = torch.LongTensor([[topi[i][0] for i in range(batch_size)]])\n",
        "            decoder_input = decoder_input.to(device)\n",
        "            # Calculate and accumulate loss\n",
        "            mask_loss, nTotal = maskNLLLoss(decoder_output, target_variable[t], mask[t])\n",
        "            loss += mask_loss\n",
        "            print_losses.append(mask_loss.item() * nTotal)\n",
        "            n_totals += nTotal\n",
        "\n",
        "    # Perform backpropatation\n",
        "    loss.backward()\n",
        "\n",
        "    # Clip gradients: gradients are modified in place\n",
        "    _ = torch.nn.utils.clip_grad_norm_(encoder.parameters(), clip)\n",
        "    _ = torch.nn.utils.clip_grad_norm_(decoder.parameters(), clip)\n",
        "\n",
        "    # Adjust model weights\n",
        "    encoder_optimizer.step()\n",
        "    decoder_optimizer.step()\n",
        "\n",
        "    return sum(print_losses) / n_totals"
      ],
      "metadata": {
        "id": "fSOQg14rXdOV"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def trainIters(model_name, voc, pairs, encoder, decoder, encoder_optimizer, decoder_optimizer, embedding, encoder_n_layers, decoder_n_layers, save_dir, n_iteration, batch_size, print_every, save_every, clip, loadFilename):\n",
        "\n",
        "    # Load batches for each iteration\n",
        "    training_batches = [batch2TrainData(voc, [random.choice(pairs) for _ in range(batch_size)])\n",
        "                      for _ in range(n_iteration)]\n",
        "\n",
        "    # Initializations\n",
        "    print('Initializing ...')\n",
        "    start_iteration = 1\n",
        "    print_loss = 0\n",
        "    if loadFilename:\n",
        "        start_iteration = checkpoint['iteration'] + 1\n",
        "\n",
        "    # Training loop\n",
        "    print(\"Training...\")\n",
        "    for iteration in range(start_iteration, n_iteration + 1):\n",
        "        training_batch = training_batches[iteration - 1]\n",
        "        # Extract fields from batch\n",
        "        input_variable, lengths, target_variable, mask, max_target_len = training_batch\n",
        "        mask = mask.bool()\n",
        "\n",
        "        # Run a training iteration with batch\n",
        "        loss = train(input_variable, lengths, target_variable, mask, max_target_len, encoder,\n",
        "                     decoder, embedding, encoder_optimizer, decoder_optimizer, batch_size, clip)\n",
        "        print_loss += loss\n",
        "\n",
        "        # Print progress\n",
        "        if iteration % print_every == 0:\n",
        "            print_loss_avg = print_loss / print_every\n",
        "            print(\"Iteration: {}; Percent complete: {:.1f}%; Average loss: {:.4f}\".format(iteration, iteration / n_iteration * 100, print_loss_avg))\n",
        "            print_loss = 0\n",
        "\n",
        "        # Save checkpoint\n",
        "        if (iteration % save_every == 0):\n",
        "            directory = os.path.join(save_dir, model_name, '{}-{}_{}'.format(encoder_n_layers, decoder_n_layers, hidden_size))\n",
        "            if not os.path.exists(directory):\n",
        "                os.makedirs(directory)\n",
        "            torch.save({\n",
        "                'iteration': iteration,\n",
        "                'en': encoder.state_dict(),\n",
        "                'de': decoder.state_dict(),\n",
        "                'en_opt': encoder_optimizer.state_dict(),\n",
        "                'de_opt': decoder_optimizer.state_dict(),\n",
        "                'loss': loss,\n",
        "                'voc_dict': voc.__dict__,\n",
        "                'embedding': embedding.state_dict()\n",
        "            }, os.path.join(directory, '{}_{}.tar'.format(iteration, 'checkpoint')))"
      ],
      "metadata": {
        "id": "rDUAqTcGXjTE"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Initialise model"
      ],
      "metadata": {
        "id": "3HQZcsVhKU8I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Configure models\n",
        "# attn_model = 'dot'\n",
        "attn_model = 'general'\n",
        "# attn_model = 'concat'\n",
        "hidden_size = 1024\n",
        "encoder_n_layers = 2\n",
        "decoder_n_layers = 2\n",
        "dropout = 0.1\n",
        "batch_size = 64\n",
        "model_name = f'lstm-{attn_model}-{encoder_n_layers}-{decoder_n_layers}-{hidden_size}'\n",
        "\n",
        "# Set checkpoint to load from; set to None if starting from scratch\n",
        "loadFilename = f\"{FOLDER_PATH}/checkpoints/{model_name}/2-2_1024/1000_checkpoint.tar\"\n",
        "# loadFilename = None\n",
        "\n",
        "# Load model if a loadFilename is provided\n",
        "if loadFilename:\n",
        "    # If loading on same machine the model was trained on\n",
        "    checkpoint = torch.load(loadFilename)\n",
        "    # If loading a model trained on GPU to CPU\n",
        "    #checkpoint = torch.load(loadFilename, map_location=torch.device('cpu'))\n",
        "    encoder_sd = checkpoint['en']\n",
        "    decoder_sd = checkpoint['de']\n",
        "    encoder_optimizer_sd = checkpoint['en_opt']\n",
        "    decoder_optimizer_sd = checkpoint['de_opt']\n",
        "    embedding_sd = checkpoint['embedding']\n",
        "    voc.__dict__ = checkpoint['voc_dict']\n",
        "\n",
        "\n",
        "print('Building encoder and decoder ...')\n",
        "# Initialize word embeddings\n",
        "embedding = nn.Embedding(voc.num_words, hidden_size)\n",
        "if loadFilename:\n",
        "    embedding.load_state_dict(embedding_sd)\n",
        "# Initialize encoder & decoder models\n",
        "encoder = EncoderRNN(hidden_size, embedding, encoder_n_layers, dropout)\n",
        "decoder = LuongAttnDecoderRNN(attn_model, embedding, hidden_size, voc.num_words, decoder_n_layers, dropout)\n",
        "if loadFilename:\n",
        "    encoder.load_state_dict(encoder_sd)\n",
        "    decoder.load_state_dict(decoder_sd)\n",
        "# Use appropriate device\n",
        "encoder = encoder.to(device)\n",
        "decoder = decoder.to(device)\n",
        "print('Models built and ready to go!')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6VG7bg-CKVCY",
        "outputId": "11136bd7-926b-44b7-e4c8-f53965b99ce7"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-16-cac9c4e6c7b6>:19: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
            "  checkpoint = torch.load(loadFilename)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Building encoder and decoder ...\n",
            "Models built and ready to go!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Run training"
      ],
      "metadata": {
        "id": "BMH_9ap-KVHD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Configure training/optimization\n",
        "clip = 50.0\n",
        "teacher_forcing_ratio = 1.0\n",
        "learning_rate = 0.0001\n",
        "decoder_learning_ratio = 5.0\n",
        "n_iteration = 1000\n",
        "print_every = 1\n",
        "save_every = 500\n",
        "\n",
        "# Ensure dropout layers are in train mode\n",
        "encoder.train()\n",
        "decoder.train()\n",
        "\n",
        "# Initialize optimizers\n",
        "print('Building optimizers ...')\n",
        "encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate)\n",
        "decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate * decoder_learning_ratio)\n",
        "if loadFilename:\n",
        "    encoder_optimizer.load_state_dict(encoder_optimizer_sd)\n",
        "    decoder_optimizer.load_state_dict(decoder_optimizer_sd)\n",
        "\n",
        "# Run training iterations\n",
        "print(\"Starting Training!\")\n",
        "save_dir = f\"{FOLDER_PATH}/checkpoints\"\n",
        "trainIters(model_name, voc, qa_pairs, encoder, decoder, encoder_optimizer, decoder_optimizer,\n",
        "           embedding, encoder_n_layers, decoder_n_layers, save_dir, n_iteration, batch_size,\n",
        "           print_every, save_every, clip, loadFilename)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VCuAeGIfKnAU",
        "outputId": "b539fac0-a444-4f0b-fcf6-0487ac33c1cd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Building optimizers ...\n",
            "Starting Training!\n",
            "Initializing ...\n",
            "Training...\n",
            "Iteration: 501; Percent complete: 50.1%; Average loss: 0.3419\n",
            "Iteration: 502; Percent complete: 50.2%; Average loss: 0.3286\n",
            "Iteration: 503; Percent complete: 50.3%; Average loss: 0.3409\n",
            "Iteration: 504; Percent complete: 50.4%; Average loss: 0.4300\n",
            "Iteration: 505; Percent complete: 50.5%; Average loss: 0.3816\n",
            "Iteration: 506; Percent complete: 50.6%; Average loss: 0.4057\n",
            "Iteration: 507; Percent complete: 50.7%; Average loss: 0.5884\n",
            "Iteration: 508; Percent complete: 50.8%; Average loss: 0.5654\n",
            "Iteration: 509; Percent complete: 50.9%; Average loss: 0.3546\n",
            "Iteration: 510; Percent complete: 51.0%; Average loss: 0.3097\n",
            "Iteration: 511; Percent complete: 51.1%; Average loss: 0.3886\n",
            "Iteration: 512; Percent complete: 51.2%; Average loss: 0.3589\n",
            "Iteration: 513; Percent complete: 51.3%; Average loss: 0.3358\n",
            "Iteration: 514; Percent complete: 51.4%; Average loss: 0.3815\n",
            "Iteration: 515; Percent complete: 51.5%; Average loss: 0.5965\n",
            "Iteration: 516; Percent complete: 51.6%; Average loss: 0.2978\n",
            "Iteration: 517; Percent complete: 51.7%; Average loss: 0.3301\n",
            "Iteration: 518; Percent complete: 51.8%; Average loss: 0.2639\n",
            "Iteration: 519; Percent complete: 51.9%; Average loss: 0.3424\n",
            "Iteration: 520; Percent complete: 52.0%; Average loss: 0.1893\n",
            "Iteration: 521; Percent complete: 52.1%; Average loss: 0.2918\n",
            "Iteration: 522; Percent complete: 52.2%; Average loss: 0.2538\n",
            "Iteration: 523; Percent complete: 52.3%; Average loss: 0.3134\n",
            "Iteration: 524; Percent complete: 52.4%; Average loss: 0.2434\n",
            "Iteration: 525; Percent complete: 52.5%; Average loss: 0.3367\n",
            "Iteration: 526; Percent complete: 52.6%; Average loss: 0.2914\n",
            "Iteration: 527; Percent complete: 52.7%; Average loss: 0.3425\n",
            "Iteration: 528; Percent complete: 52.8%; Average loss: 0.2789\n",
            "Iteration: 529; Percent complete: 52.9%; Average loss: 0.3890\n",
            "Iteration: 530; Percent complete: 53.0%; Average loss: 0.3689\n",
            "Iteration: 531; Percent complete: 53.1%; Average loss: 0.3261\n",
            "Iteration: 532; Percent complete: 53.2%; Average loss: 0.3464\n",
            "Iteration: 533; Percent complete: 53.3%; Average loss: 0.3156\n",
            "Iteration: 534; Percent complete: 53.4%; Average loss: 0.3813\n",
            "Iteration: 535; Percent complete: 53.5%; Average loss: 0.3355\n",
            "Iteration: 536; Percent complete: 53.6%; Average loss: 0.1759\n",
            "Iteration: 537; Percent complete: 53.7%; Average loss: 0.3209\n",
            "Iteration: 538; Percent complete: 53.8%; Average loss: 0.1982\n",
            "Iteration: 539; Percent complete: 53.9%; Average loss: 0.2955\n",
            "Iteration: 540; Percent complete: 54.0%; Average loss: 0.2909\n",
            "Iteration: 541; Percent complete: 54.1%; Average loss: 0.2574\n",
            "Iteration: 542; Percent complete: 54.2%; Average loss: 0.3315\n",
            "Iteration: 543; Percent complete: 54.3%; Average loss: 0.3195\n",
            "Iteration: 544; Percent complete: 54.4%; Average loss: 0.2803\n",
            "Iteration: 545; Percent complete: 54.5%; Average loss: 0.4305\n",
            "Iteration: 546; Percent complete: 54.6%; Average loss: 0.2445\n",
            "Iteration: 547; Percent complete: 54.7%; Average loss: 0.2851\n",
            "Iteration: 548; Percent complete: 54.8%; Average loss: 0.2518\n",
            "Iteration: 549; Percent complete: 54.9%; Average loss: 0.2548\n",
            "Iteration: 550; Percent complete: 55.0%; Average loss: 0.2431\n",
            "Iteration: 551; Percent complete: 55.1%; Average loss: 0.2492\n",
            "Iteration: 552; Percent complete: 55.2%; Average loss: 0.2478\n",
            "Iteration: 553; Percent complete: 55.3%; Average loss: 0.3254\n",
            "Iteration: 554; Percent complete: 55.4%; Average loss: 0.3009\n",
            "Iteration: 555; Percent complete: 55.5%; Average loss: 0.2729\n",
            "Iteration: 556; Percent complete: 55.6%; Average loss: 0.3228\n",
            "Iteration: 557; Percent complete: 55.7%; Average loss: 0.2925\n",
            "Iteration: 558; Percent complete: 55.8%; Average loss: 0.1926\n",
            "Iteration: 559; Percent complete: 55.9%; Average loss: 0.2288\n",
            "Iteration: 560; Percent complete: 56.0%; Average loss: 0.2449\n",
            "Iteration: 561; Percent complete: 56.1%; Average loss: 0.2617\n",
            "Iteration: 562; Percent complete: 56.2%; Average loss: 0.2184\n",
            "Iteration: 563; Percent complete: 56.3%; Average loss: 0.2667\n",
            "Iteration: 564; Percent complete: 56.4%; Average loss: 0.2647\n",
            "Iteration: 565; Percent complete: 56.5%; Average loss: 0.2803\n",
            "Iteration: 566; Percent complete: 56.6%; Average loss: 0.3194\n",
            "Iteration: 567; Percent complete: 56.7%; Average loss: 0.2122\n",
            "Iteration: 568; Percent complete: 56.8%; Average loss: 0.2612\n",
            "Iteration: 569; Percent complete: 56.9%; Average loss: 0.3237\n",
            "Iteration: 570; Percent complete: 57.0%; Average loss: 0.3067\n",
            "Iteration: 571; Percent complete: 57.1%; Average loss: 0.2404\n",
            "Iteration: 572; Percent complete: 57.2%; Average loss: 0.2469\n",
            "Iteration: 573; Percent complete: 57.3%; Average loss: 0.2457\n",
            "Iteration: 574; Percent complete: 57.4%; Average loss: 0.2433\n",
            "Iteration: 575; Percent complete: 57.5%; Average loss: 0.3148\n",
            "Iteration: 576; Percent complete: 57.6%; Average loss: 0.2484\n",
            "Iteration: 577; Percent complete: 57.7%; Average loss: 0.2736\n",
            "Iteration: 578; Percent complete: 57.8%; Average loss: 0.1817\n",
            "Iteration: 579; Percent complete: 57.9%; Average loss: 0.2285\n",
            "Iteration: 580; Percent complete: 58.0%; Average loss: 0.2547\n",
            "Iteration: 581; Percent complete: 58.1%; Average loss: 0.2263\n",
            "Iteration: 582; Percent complete: 58.2%; Average loss: 0.2454\n",
            "Iteration: 583; Percent complete: 58.3%; Average loss: 0.1782\n",
            "Iteration: 584; Percent complete: 58.4%; Average loss: 0.1888\n",
            "Iteration: 585; Percent complete: 58.5%; Average loss: 0.1920\n",
            "Iteration: 586; Percent complete: 58.6%; Average loss: 0.2537\n",
            "Iteration: 587; Percent complete: 58.7%; Average loss: 0.2393\n",
            "Iteration: 588; Percent complete: 58.8%; Average loss: 0.3423\n",
            "Iteration: 589; Percent complete: 58.9%; Average loss: 0.2596\n",
            "Iteration: 590; Percent complete: 59.0%; Average loss: 0.1495\n",
            "Iteration: 591; Percent complete: 59.1%; Average loss: 0.2147\n",
            "Iteration: 592; Percent complete: 59.2%; Average loss: 0.2440\n",
            "Iteration: 593; Percent complete: 59.3%; Average loss: 0.2292\n",
            "Iteration: 594; Percent complete: 59.4%; Average loss: 0.1929\n",
            "Iteration: 595; Percent complete: 59.5%; Average loss: 0.2831\n",
            "Iteration: 596; Percent complete: 59.6%; Average loss: 0.1806\n",
            "Iteration: 597; Percent complete: 59.7%; Average loss: 0.1698\n",
            "Iteration: 598; Percent complete: 59.8%; Average loss: 0.1429\n",
            "Iteration: 599; Percent complete: 59.9%; Average loss: 0.2651\n",
            "Iteration: 600; Percent complete: 60.0%; Average loss: 0.2424\n",
            "Iteration: 601; Percent complete: 60.1%; Average loss: 0.1496\n",
            "Iteration: 602; Percent complete: 60.2%; Average loss: 0.1794\n",
            "Iteration: 603; Percent complete: 60.3%; Average loss: 0.1886\n",
            "Iteration: 604; Percent complete: 60.4%; Average loss: 0.2218\n",
            "Iteration: 605; Percent complete: 60.5%; Average loss: 0.1811\n",
            "Iteration: 606; Percent complete: 60.6%; Average loss: 0.2308\n",
            "Iteration: 607; Percent complete: 60.7%; Average loss: 0.2201\n",
            "Iteration: 608; Percent complete: 60.8%; Average loss: 0.2950\n",
            "Iteration: 609; Percent complete: 60.9%; Average loss: 0.2070\n",
            "Iteration: 610; Percent complete: 61.0%; Average loss: 0.2154\n",
            "Iteration: 611; Percent complete: 61.1%; Average loss: 0.1915\n",
            "Iteration: 612; Percent complete: 61.2%; Average loss: 0.1577\n",
            "Iteration: 613; Percent complete: 61.3%; Average loss: 0.3495\n",
            "Iteration: 614; Percent complete: 61.4%; Average loss: 0.1677\n",
            "Iteration: 615; Percent complete: 61.5%; Average loss: 0.1756\n",
            "Iteration: 616; Percent complete: 61.6%; Average loss: 0.1912\n",
            "Iteration: 617; Percent complete: 61.7%; Average loss: 0.1916\n",
            "Iteration: 618; Percent complete: 61.8%; Average loss: 0.1764\n",
            "Iteration: 619; Percent complete: 61.9%; Average loss: 0.2635\n",
            "Iteration: 620; Percent complete: 62.0%; Average loss: 0.1687\n",
            "Iteration: 621; Percent complete: 62.1%; Average loss: 0.1403\n",
            "Iteration: 622; Percent complete: 62.2%; Average loss: 0.1848\n",
            "Iteration: 623; Percent complete: 62.3%; Average loss: 0.1983\n",
            "Iteration: 624; Percent complete: 62.4%; Average loss: 0.1725\n",
            "Iteration: 625; Percent complete: 62.5%; Average loss: 0.1433\n",
            "Iteration: 626; Percent complete: 62.6%; Average loss: 0.1438\n",
            "Iteration: 627; Percent complete: 62.7%; Average loss: 0.1422\n",
            "Iteration: 628; Percent complete: 62.8%; Average loss: 0.1861\n",
            "Iteration: 629; Percent complete: 62.9%; Average loss: 0.1683\n",
            "Iteration: 630; Percent complete: 63.0%; Average loss: 0.1621\n",
            "Iteration: 631; Percent complete: 63.1%; Average loss: 0.1213\n",
            "Iteration: 632; Percent complete: 63.2%; Average loss: 0.1853\n",
            "Iteration: 633; Percent complete: 63.3%; Average loss: 0.1959\n",
            "Iteration: 634; Percent complete: 63.4%; Average loss: 0.1870\n",
            "Iteration: 635; Percent complete: 63.5%; Average loss: 0.1653\n",
            "Iteration: 636; Percent complete: 63.6%; Average loss: 0.1944\n",
            "Iteration: 637; Percent complete: 63.7%; Average loss: 0.1886\n",
            "Iteration: 638; Percent complete: 63.8%; Average loss: 0.1606\n",
            "Iteration: 639; Percent complete: 63.9%; Average loss: 0.1782\n",
            "Iteration: 640; Percent complete: 64.0%; Average loss: 0.1388\n",
            "Iteration: 641; Percent complete: 64.1%; Average loss: 0.1833\n",
            "Iteration: 642; Percent complete: 64.2%; Average loss: 0.1662\n",
            "Iteration: 643; Percent complete: 64.3%; Average loss: 0.1507\n",
            "Iteration: 644; Percent complete: 64.4%; Average loss: 0.2012\n",
            "Iteration: 645; Percent complete: 64.5%; Average loss: 0.1428\n",
            "Iteration: 646; Percent complete: 64.6%; Average loss: 0.1753\n",
            "Iteration: 647; Percent complete: 64.7%; Average loss: 0.1376\n",
            "Iteration: 648; Percent complete: 64.8%; Average loss: 0.2345\n",
            "Iteration: 649; Percent complete: 64.9%; Average loss: 0.1341\n",
            "Iteration: 650; Percent complete: 65.0%; Average loss: 0.1521\n",
            "Iteration: 651; Percent complete: 65.1%; Average loss: 0.2061\n",
            "Iteration: 652; Percent complete: 65.2%; Average loss: 0.1247\n",
            "Iteration: 653; Percent complete: 65.3%; Average loss: 0.1714\n",
            "Iteration: 654; Percent complete: 65.4%; Average loss: 0.1350\n",
            "Iteration: 655; Percent complete: 65.5%; Average loss: 0.0916\n",
            "Iteration: 656; Percent complete: 65.6%; Average loss: 0.1622\n",
            "Iteration: 657; Percent complete: 65.7%; Average loss: 0.1812\n",
            "Iteration: 658; Percent complete: 65.8%; Average loss: 0.1096\n",
            "Iteration: 659; Percent complete: 65.9%; Average loss: 0.1241\n",
            "Iteration: 660; Percent complete: 66.0%; Average loss: 0.1670\n",
            "Iteration: 661; Percent complete: 66.1%; Average loss: 0.1806\n",
            "Iteration: 662; Percent complete: 66.2%; Average loss: 0.1049\n",
            "Iteration: 663; Percent complete: 66.3%; Average loss: 0.1216\n",
            "Iteration: 664; Percent complete: 66.4%; Average loss: 0.1208\n",
            "Iteration: 665; Percent complete: 66.5%; Average loss: 0.1350\n",
            "Iteration: 666; Percent complete: 66.6%; Average loss: 0.1008\n",
            "Iteration: 667; Percent complete: 66.7%; Average loss: 0.1660\n",
            "Iteration: 668; Percent complete: 66.8%; Average loss: 0.1439\n",
            "Iteration: 669; Percent complete: 66.9%; Average loss: 0.1907\n",
            "Iteration: 670; Percent complete: 67.0%; Average loss: 0.1283\n",
            "Iteration: 671; Percent complete: 67.1%; Average loss: 0.1592\n",
            "Iteration: 672; Percent complete: 67.2%; Average loss: 0.0976\n",
            "Iteration: 673; Percent complete: 67.3%; Average loss: 0.1345\n",
            "Iteration: 674; Percent complete: 67.4%; Average loss: 0.1237\n",
            "Iteration: 675; Percent complete: 67.5%; Average loss: 0.1399\n",
            "Iteration: 676; Percent complete: 67.6%; Average loss: 0.1294\n",
            "Iteration: 677; Percent complete: 67.7%; Average loss: 0.1060\n",
            "Iteration: 678; Percent complete: 67.8%; Average loss: 0.1502\n",
            "Iteration: 679; Percent complete: 67.9%; Average loss: 0.1201\n",
            "Iteration: 680; Percent complete: 68.0%; Average loss: 0.1135\n",
            "Iteration: 681; Percent complete: 68.1%; Average loss: 0.1272\n",
            "Iteration: 682; Percent complete: 68.2%; Average loss: 0.1357\n",
            "Iteration: 683; Percent complete: 68.3%; Average loss: 0.1318\n",
            "Iteration: 684; Percent complete: 68.4%; Average loss: 0.1844\n",
            "Iteration: 685; Percent complete: 68.5%; Average loss: 0.1233\n",
            "Iteration: 686; Percent complete: 68.6%; Average loss: 0.1360\n",
            "Iteration: 687; Percent complete: 68.7%; Average loss: 0.1069\n",
            "Iteration: 688; Percent complete: 68.8%; Average loss: 0.1321\n",
            "Iteration: 689; Percent complete: 68.9%; Average loss: 0.1336\n",
            "Iteration: 690; Percent complete: 69.0%; Average loss: 0.0823\n",
            "Iteration: 691; Percent complete: 69.1%; Average loss: 0.1602\n",
            "Iteration: 692; Percent complete: 69.2%; Average loss: 0.1161\n",
            "Iteration: 693; Percent complete: 69.3%; Average loss: 0.1421\n",
            "Iteration: 694; Percent complete: 69.4%; Average loss: 0.1883\n",
            "Iteration: 695; Percent complete: 69.5%; Average loss: 0.1484\n",
            "Iteration: 696; Percent complete: 69.6%; Average loss: 0.1193\n",
            "Iteration: 697; Percent complete: 69.7%; Average loss: 0.1388\n",
            "Iteration: 698; Percent complete: 69.8%; Average loss: 0.1498\n",
            "Iteration: 699; Percent complete: 69.9%; Average loss: 0.1182\n",
            "Iteration: 700; Percent complete: 70.0%; Average loss: 0.0995\n",
            "Iteration: 701; Percent complete: 70.1%; Average loss: 0.1244\n",
            "Iteration: 702; Percent complete: 70.2%; Average loss: 0.1166\n",
            "Iteration: 703; Percent complete: 70.3%; Average loss: 0.1320\n",
            "Iteration: 704; Percent complete: 70.4%; Average loss: 0.1604\n",
            "Iteration: 705; Percent complete: 70.5%; Average loss: 0.0708\n",
            "Iteration: 706; Percent complete: 70.6%; Average loss: 0.0934\n",
            "Iteration: 707; Percent complete: 70.7%; Average loss: 0.1159\n",
            "Iteration: 708; Percent complete: 70.8%; Average loss: 0.1567\n",
            "Iteration: 709; Percent complete: 70.9%; Average loss: 0.1093\n",
            "Iteration: 710; Percent complete: 71.0%; Average loss: 0.1339\n",
            "Iteration: 711; Percent complete: 71.1%; Average loss: 0.1316\n",
            "Iteration: 712; Percent complete: 71.2%; Average loss: 0.1378\n",
            "Iteration: 713; Percent complete: 71.3%; Average loss: 0.1126\n",
            "Iteration: 714; Percent complete: 71.4%; Average loss: 0.1179\n",
            "Iteration: 715; Percent complete: 71.5%; Average loss: 0.1231\n",
            "Iteration: 716; Percent complete: 71.6%; Average loss: 0.1257\n",
            "Iteration: 717; Percent complete: 71.7%; Average loss: 0.1067\n",
            "Iteration: 718; Percent complete: 71.8%; Average loss: 0.1097\n",
            "Iteration: 719; Percent complete: 71.9%; Average loss: 0.0908\n",
            "Iteration: 720; Percent complete: 72.0%; Average loss: 0.0932\n",
            "Iteration: 721; Percent complete: 72.1%; Average loss: 0.1392\n",
            "Iteration: 722; Percent complete: 72.2%; Average loss: 0.1077\n",
            "Iteration: 723; Percent complete: 72.3%; Average loss: 0.0859\n",
            "Iteration: 724; Percent complete: 72.4%; Average loss: 0.0944\n",
            "Iteration: 725; Percent complete: 72.5%; Average loss: 0.1427\n",
            "Iteration: 726; Percent complete: 72.6%; Average loss: 0.1238\n",
            "Iteration: 727; Percent complete: 72.7%; Average loss: 0.1241\n",
            "Iteration: 728; Percent complete: 72.8%; Average loss: 0.1155\n",
            "Iteration: 729; Percent complete: 72.9%; Average loss: 0.0794\n",
            "Iteration: 730; Percent complete: 73.0%; Average loss: 0.1376\n",
            "Iteration: 731; Percent complete: 73.1%; Average loss: 0.1269\n",
            "Iteration: 732; Percent complete: 73.2%; Average loss: 0.1026\n",
            "Iteration: 733; Percent complete: 73.3%; Average loss: 0.0700\n",
            "Iteration: 734; Percent complete: 73.4%; Average loss: 0.1010\n",
            "Iteration: 735; Percent complete: 73.5%; Average loss: 0.0693\n",
            "Iteration: 736; Percent complete: 73.6%; Average loss: 0.1012\n",
            "Iteration: 737; Percent complete: 73.7%; Average loss: 0.1115\n",
            "Iteration: 738; Percent complete: 73.8%; Average loss: 0.0971\n",
            "Iteration: 739; Percent complete: 73.9%; Average loss: 0.0918\n",
            "Iteration: 740; Percent complete: 74.0%; Average loss: 0.0905\n",
            "Iteration: 741; Percent complete: 74.1%; Average loss: 0.1026\n",
            "Iteration: 742; Percent complete: 74.2%; Average loss: 0.0907\n",
            "Iteration: 743; Percent complete: 74.3%; Average loss: 0.1184\n",
            "Iteration: 744; Percent complete: 74.4%; Average loss: 0.0929\n",
            "Iteration: 745; Percent complete: 74.5%; Average loss: 0.1089\n",
            "Iteration: 746; Percent complete: 74.6%; Average loss: 0.1006\n",
            "Iteration: 747; Percent complete: 74.7%; Average loss: 0.1038\n",
            "Iteration: 748; Percent complete: 74.8%; Average loss: 0.0943\n",
            "Iteration: 749; Percent complete: 74.9%; Average loss: 0.0792\n",
            "Iteration: 750; Percent complete: 75.0%; Average loss: 0.1009\n",
            "Iteration: 751; Percent complete: 75.1%; Average loss: 0.1123\n",
            "Iteration: 752; Percent complete: 75.2%; Average loss: 0.0944\n",
            "Iteration: 753; Percent complete: 75.3%; Average loss: 0.0968\n",
            "Iteration: 754; Percent complete: 75.4%; Average loss: 0.0950\n",
            "Iteration: 755; Percent complete: 75.5%; Average loss: 0.0779\n",
            "Iteration: 756; Percent complete: 75.6%; Average loss: 0.1228\n",
            "Iteration: 757; Percent complete: 75.7%; Average loss: 0.0905\n",
            "Iteration: 758; Percent complete: 75.8%; Average loss: 0.1001\n",
            "Iteration: 759; Percent complete: 75.9%; Average loss: 0.0820\n",
            "Iteration: 760; Percent complete: 76.0%; Average loss: 0.1560\n",
            "Iteration: 761; Percent complete: 76.1%; Average loss: 0.1049\n",
            "Iteration: 762; Percent complete: 76.2%; Average loss: 0.0746\n",
            "Iteration: 763; Percent complete: 76.3%; Average loss: 0.1057\n",
            "Iteration: 764; Percent complete: 76.4%; Average loss: 0.0952\n",
            "Iteration: 765; Percent complete: 76.5%; Average loss: 0.0637\n",
            "Iteration: 766; Percent complete: 76.6%; Average loss: 0.0972\n",
            "Iteration: 767; Percent complete: 76.7%; Average loss: 0.0966\n",
            "Iteration: 768; Percent complete: 76.8%; Average loss: 0.0884\n",
            "Iteration: 769; Percent complete: 76.9%; Average loss: 0.0842\n",
            "Iteration: 770; Percent complete: 77.0%; Average loss: 0.0793\n",
            "Iteration: 771; Percent complete: 77.1%; Average loss: 0.1129\n",
            "Iteration: 772; Percent complete: 77.2%; Average loss: 0.0472\n",
            "Iteration: 773; Percent complete: 77.3%; Average loss: 0.0657\n",
            "Iteration: 774; Percent complete: 77.4%; Average loss: 0.1046\n",
            "Iteration: 775; Percent complete: 77.5%; Average loss: 0.0676\n",
            "Iteration: 776; Percent complete: 77.6%; Average loss: 0.0686\n",
            "Iteration: 777; Percent complete: 77.7%; Average loss: 0.0830\n",
            "Iteration: 778; Percent complete: 77.8%; Average loss: 0.0874\n",
            "Iteration: 779; Percent complete: 77.9%; Average loss: 0.0857\n",
            "Iteration: 780; Percent complete: 78.0%; Average loss: 0.0692\n",
            "Iteration: 781; Percent complete: 78.1%; Average loss: 0.0894\n",
            "Iteration: 782; Percent complete: 78.2%; Average loss: 0.0852\n",
            "Iteration: 783; Percent complete: 78.3%; Average loss: 0.0556\n",
            "Iteration: 784; Percent complete: 78.4%; Average loss: 0.0773\n",
            "Iteration: 785; Percent complete: 78.5%; Average loss: 0.0727\n",
            "Iteration: 786; Percent complete: 78.6%; Average loss: 0.0474\n",
            "Iteration: 787; Percent complete: 78.7%; Average loss: 0.0684\n",
            "Iteration: 788; Percent complete: 78.8%; Average loss: 0.0637\n",
            "Iteration: 789; Percent complete: 78.9%; Average loss: 0.0494\n",
            "Iteration: 790; Percent complete: 79.0%; Average loss: 0.0866\n",
            "Iteration: 791; Percent complete: 79.1%; Average loss: 0.0769\n",
            "Iteration: 792; Percent complete: 79.2%; Average loss: 0.1157\n",
            "Iteration: 793; Percent complete: 79.3%; Average loss: 0.0736\n",
            "Iteration: 794; Percent complete: 79.4%; Average loss: 0.0628\n",
            "Iteration: 795; Percent complete: 79.5%; Average loss: 0.0759\n",
            "Iteration: 796; Percent complete: 79.6%; Average loss: 0.0486\n",
            "Iteration: 797; Percent complete: 79.7%; Average loss: 0.0500\n",
            "Iteration: 798; Percent complete: 79.8%; Average loss: 0.0802\n",
            "Iteration: 799; Percent complete: 79.9%; Average loss: 0.1180\n",
            "Iteration: 800; Percent complete: 80.0%; Average loss: 0.0577\n",
            "Iteration: 801; Percent complete: 80.1%; Average loss: 0.0818\n",
            "Iteration: 802; Percent complete: 80.2%; Average loss: 0.0790\n",
            "Iteration: 803; Percent complete: 80.3%; Average loss: 0.0696\n",
            "Iteration: 804; Percent complete: 80.4%; Average loss: 0.0518\n",
            "Iteration: 805; Percent complete: 80.5%; Average loss: 0.0483\n",
            "Iteration: 806; Percent complete: 80.6%; Average loss: 0.0668\n",
            "Iteration: 807; Percent complete: 80.7%; Average loss: 0.0693\n",
            "Iteration: 808; Percent complete: 80.8%; Average loss: 0.0671\n",
            "Iteration: 809; Percent complete: 80.9%; Average loss: 0.0706\n",
            "Iteration: 810; Percent complete: 81.0%; Average loss: 0.0375\n",
            "Iteration: 811; Percent complete: 81.1%; Average loss: 0.0761\n",
            "Iteration: 812; Percent complete: 81.2%; Average loss: 0.0633\n",
            "Iteration: 813; Percent complete: 81.3%; Average loss: 0.0897\n",
            "Iteration: 814; Percent complete: 81.4%; Average loss: 0.0535\n",
            "Iteration: 815; Percent complete: 81.5%; Average loss: 0.0869\n",
            "Iteration: 816; Percent complete: 81.6%; Average loss: 0.1024\n",
            "Iteration: 817; Percent complete: 81.7%; Average loss: 0.0488\n",
            "Iteration: 818; Percent complete: 81.8%; Average loss: 0.0513\n",
            "Iteration: 819; Percent complete: 81.9%; Average loss: 0.0869\n",
            "Iteration: 820; Percent complete: 82.0%; Average loss: 0.0646\n",
            "Iteration: 821; Percent complete: 82.1%; Average loss: 0.0664\n",
            "Iteration: 822; Percent complete: 82.2%; Average loss: 0.0607\n",
            "Iteration: 823; Percent complete: 82.3%; Average loss: 0.0475\n",
            "Iteration: 824; Percent complete: 82.4%; Average loss: 0.0664\n",
            "Iteration: 825; Percent complete: 82.5%; Average loss: 0.0578\n",
            "Iteration: 826; Percent complete: 82.6%; Average loss: 0.0662\n",
            "Iteration: 827; Percent complete: 82.7%; Average loss: 0.0619\n",
            "Iteration: 828; Percent complete: 82.8%; Average loss: 0.0708\n",
            "Iteration: 829; Percent complete: 82.9%; Average loss: 0.0461\n",
            "Iteration: 830; Percent complete: 83.0%; Average loss: 0.0552\n",
            "Iteration: 831; Percent complete: 83.1%; Average loss: 0.0653\n",
            "Iteration: 832; Percent complete: 83.2%; Average loss: 0.0585\n",
            "Iteration: 833; Percent complete: 83.3%; Average loss: 0.1030\n",
            "Iteration: 834; Percent complete: 83.4%; Average loss: 0.0483\n",
            "Iteration: 835; Percent complete: 83.5%; Average loss: 0.0617\n",
            "Iteration: 836; Percent complete: 83.6%; Average loss: 0.0561\n",
            "Iteration: 837; Percent complete: 83.7%; Average loss: 0.0533\n",
            "Iteration: 838; Percent complete: 83.8%; Average loss: 0.0622\n",
            "Iteration: 839; Percent complete: 83.9%; Average loss: 0.0515\n",
            "Iteration: 840; Percent complete: 84.0%; Average loss: 0.0613\n",
            "Iteration: 841; Percent complete: 84.1%; Average loss: 0.0600\n",
            "Iteration: 842; Percent complete: 84.2%; Average loss: 0.0665\n",
            "Iteration: 843; Percent complete: 84.3%; Average loss: 0.0537\n",
            "Iteration: 844; Percent complete: 84.4%; Average loss: 0.0582\n",
            "Iteration: 845; Percent complete: 84.5%; Average loss: 0.0379\n",
            "Iteration: 846; Percent complete: 84.6%; Average loss: 0.0586\n",
            "Iteration: 847; Percent complete: 84.7%; Average loss: 0.0529\n",
            "Iteration: 848; Percent complete: 84.8%; Average loss: 0.0610\n",
            "Iteration: 849; Percent complete: 84.9%; Average loss: 0.0318\n",
            "Iteration: 850; Percent complete: 85.0%; Average loss: 0.0582\n",
            "Iteration: 851; Percent complete: 85.1%; Average loss: 0.0448\n",
            "Iteration: 852; Percent complete: 85.2%; Average loss: 0.0580\n",
            "Iteration: 853; Percent complete: 85.3%; Average loss: 0.0579\n",
            "Iteration: 854; Percent complete: 85.4%; Average loss: 0.0746\n",
            "Iteration: 855; Percent complete: 85.5%; Average loss: 0.0541\n",
            "Iteration: 856; Percent complete: 85.6%; Average loss: 0.0573\n",
            "Iteration: 857; Percent complete: 85.7%; Average loss: 0.0570\n",
            "Iteration: 858; Percent complete: 85.8%; Average loss: 0.0531\n",
            "Iteration: 859; Percent complete: 85.9%; Average loss: 0.0439\n",
            "Iteration: 860; Percent complete: 86.0%; Average loss: 0.0391\n",
            "Iteration: 861; Percent complete: 86.1%; Average loss: 0.0544\n",
            "Iteration: 862; Percent complete: 86.2%; Average loss: 0.0589\n",
            "Iteration: 863; Percent complete: 86.3%; Average loss: 0.0470\n",
            "Iteration: 864; Percent complete: 86.4%; Average loss: 0.0418\n",
            "Iteration: 865; Percent complete: 86.5%; Average loss: 0.0640\n",
            "Iteration: 866; Percent complete: 86.6%; Average loss: 0.0629\n",
            "Iteration: 867; Percent complete: 86.7%; Average loss: 0.0707\n",
            "Iteration: 868; Percent complete: 86.8%; Average loss: 0.0453\n",
            "Iteration: 869; Percent complete: 86.9%; Average loss: 0.0396\n",
            "Iteration: 870; Percent complete: 87.0%; Average loss: 0.0395\n",
            "Iteration: 871; Percent complete: 87.1%; Average loss: 0.0508\n",
            "Iteration: 872; Percent complete: 87.2%; Average loss: 0.0685\n",
            "Iteration: 873; Percent complete: 87.3%; Average loss: 0.0620\n",
            "Iteration: 874; Percent complete: 87.4%; Average loss: 0.0318\n",
            "Iteration: 875; Percent complete: 87.5%; Average loss: 0.0375\n",
            "Iteration: 876; Percent complete: 87.6%; Average loss: 0.0488\n",
            "Iteration: 877; Percent complete: 87.7%; Average loss: 0.0498\n",
            "Iteration: 878; Percent complete: 87.8%; Average loss: 0.0584\n",
            "Iteration: 879; Percent complete: 87.9%; Average loss: 0.0849\n",
            "Iteration: 880; Percent complete: 88.0%; Average loss: 0.0405\n",
            "Iteration: 881; Percent complete: 88.1%; Average loss: 0.0323\n",
            "Iteration: 882; Percent complete: 88.2%; Average loss: 0.0559\n",
            "Iteration: 883; Percent complete: 88.3%; Average loss: 0.0459\n",
            "Iteration: 884; Percent complete: 88.4%; Average loss: 0.0722\n",
            "Iteration: 885; Percent complete: 88.5%; Average loss: 0.0386\n",
            "Iteration: 886; Percent complete: 88.6%; Average loss: 0.0511\n",
            "Iteration: 887; Percent complete: 88.7%; Average loss: 0.0442\n",
            "Iteration: 888; Percent complete: 88.8%; Average loss: 0.0396\n",
            "Iteration: 889; Percent complete: 88.9%; Average loss: 0.0366\n",
            "Iteration: 890; Percent complete: 89.0%; Average loss: 0.0286\n",
            "Iteration: 891; Percent complete: 89.1%; Average loss: 0.0473\n",
            "Iteration: 892; Percent complete: 89.2%; Average loss: 0.0539\n",
            "Iteration: 893; Percent complete: 89.3%; Average loss: 0.0440\n",
            "Iteration: 894; Percent complete: 89.4%; Average loss: 0.0360\n",
            "Iteration: 895; Percent complete: 89.5%; Average loss: 0.0376\n",
            "Iteration: 896; Percent complete: 89.6%; Average loss: 0.0693\n",
            "Iteration: 897; Percent complete: 89.7%; Average loss: 0.0480\n",
            "Iteration: 898; Percent complete: 89.8%; Average loss: 0.0581\n",
            "Iteration: 899; Percent complete: 89.9%; Average loss: 0.0848\n",
            "Iteration: 900; Percent complete: 90.0%; Average loss: 0.0543\n",
            "Iteration: 901; Percent complete: 90.1%; Average loss: 0.0510\n",
            "Iteration: 902; Percent complete: 90.2%; Average loss: 0.0463\n",
            "Iteration: 903; Percent complete: 90.3%; Average loss: 0.0465\n",
            "Iteration: 904; Percent complete: 90.4%; Average loss: 0.0350\n",
            "Iteration: 905; Percent complete: 90.5%; Average loss: 0.0341\n",
            "Iteration: 906; Percent complete: 90.6%; Average loss: 0.0496\n",
            "Iteration: 907; Percent complete: 90.7%; Average loss: 0.0356\n",
            "Iteration: 908; Percent complete: 90.8%; Average loss: 0.0506\n",
            "Iteration: 909; Percent complete: 90.9%; Average loss: 0.0412\n",
            "Iteration: 910; Percent complete: 91.0%; Average loss: 0.0298\n",
            "Iteration: 911; Percent complete: 91.1%; Average loss: 0.0489\n",
            "Iteration: 912; Percent complete: 91.2%; Average loss: 0.0367\n",
            "Iteration: 913; Percent complete: 91.3%; Average loss: 0.0490\n",
            "Iteration: 914; Percent complete: 91.4%; Average loss: 0.0445\n",
            "Iteration: 915; Percent complete: 91.5%; Average loss: 0.0436\n",
            "Iteration: 916; Percent complete: 91.6%; Average loss: 0.0754\n",
            "Iteration: 917; Percent complete: 91.7%; Average loss: 0.0440\n",
            "Iteration: 918; Percent complete: 91.8%; Average loss: 0.0524\n",
            "Iteration: 919; Percent complete: 91.9%; Average loss: 0.0477\n",
            "Iteration: 920; Percent complete: 92.0%; Average loss: 0.0328\n",
            "Iteration: 921; Percent complete: 92.1%; Average loss: 0.0294\n",
            "Iteration: 922; Percent complete: 92.2%; Average loss: 0.0319\n",
            "Iteration: 923; Percent complete: 92.3%; Average loss: 0.0402\n",
            "Iteration: 924; Percent complete: 92.4%; Average loss: 0.0345\n",
            "Iteration: 925; Percent complete: 92.5%; Average loss: 0.0357\n",
            "Iteration: 926; Percent complete: 92.6%; Average loss: 0.0336\n",
            "Iteration: 927; Percent complete: 92.7%; Average loss: 0.0467\n",
            "Iteration: 928; Percent complete: 92.8%; Average loss: 0.0388\n",
            "Iteration: 929; Percent complete: 92.9%; Average loss: 0.0328\n",
            "Iteration: 930; Percent complete: 93.0%; Average loss: 0.0551\n",
            "Iteration: 931; Percent complete: 93.1%; Average loss: 0.0469\n",
            "Iteration: 932; Percent complete: 93.2%; Average loss: 0.0443\n",
            "Iteration: 933; Percent complete: 93.3%; Average loss: 0.0446\n",
            "Iteration: 934; Percent complete: 93.4%; Average loss: 0.0458\n",
            "Iteration: 935; Percent complete: 93.5%; Average loss: 0.0206\n",
            "Iteration: 936; Percent complete: 93.6%; Average loss: 0.0667\n",
            "Iteration: 937; Percent complete: 93.7%; Average loss: 0.0616\n",
            "Iteration: 938; Percent complete: 93.8%; Average loss: 0.0365\n",
            "Iteration: 939; Percent complete: 93.9%; Average loss: 0.0437\n",
            "Iteration: 940; Percent complete: 94.0%; Average loss: 0.0521\n",
            "Iteration: 941; Percent complete: 94.1%; Average loss: 0.0398\n",
            "Iteration: 942; Percent complete: 94.2%; Average loss: 0.0310\n",
            "Iteration: 943; Percent complete: 94.3%; Average loss: 0.0586\n",
            "Iteration: 944; Percent complete: 94.4%; Average loss: 0.0554\n",
            "Iteration: 945; Percent complete: 94.5%; Average loss: 0.0393\n",
            "Iteration: 946; Percent complete: 94.6%; Average loss: 0.0366\n",
            "Iteration: 947; Percent complete: 94.7%; Average loss: 0.0290\n",
            "Iteration: 948; Percent complete: 94.8%; Average loss: 0.0438\n",
            "Iteration: 949; Percent complete: 94.9%; Average loss: 0.0269\n",
            "Iteration: 950; Percent complete: 95.0%; Average loss: 0.0436\n",
            "Iteration: 951; Percent complete: 95.1%; Average loss: 0.0357\n",
            "Iteration: 952; Percent complete: 95.2%; Average loss: 0.0315\n",
            "Iteration: 953; Percent complete: 95.3%; Average loss: 0.0386\n",
            "Iteration: 954; Percent complete: 95.4%; Average loss: 0.0448\n",
            "Iteration: 955; Percent complete: 95.5%; Average loss: 0.0371\n",
            "Iteration: 956; Percent complete: 95.6%; Average loss: 0.0401\n",
            "Iteration: 957; Percent complete: 95.7%; Average loss: 0.0476\n",
            "Iteration: 958; Percent complete: 95.8%; Average loss: 0.0358\n",
            "Iteration: 959; Percent complete: 95.9%; Average loss: 0.0291\n",
            "Iteration: 960; Percent complete: 96.0%; Average loss: 0.0339\n",
            "Iteration: 961; Percent complete: 96.1%; Average loss: 0.0427\n",
            "Iteration: 962; Percent complete: 96.2%; Average loss: 0.0303\n",
            "Iteration: 963; Percent complete: 96.3%; Average loss: 0.0357\n",
            "Iteration: 964; Percent complete: 96.4%; Average loss: 0.0349\n",
            "Iteration: 965; Percent complete: 96.5%; Average loss: 0.0514\n",
            "Iteration: 966; Percent complete: 96.6%; Average loss: 0.0306\n",
            "Iteration: 967; Percent complete: 96.7%; Average loss: 0.0364\n",
            "Iteration: 968; Percent complete: 96.8%; Average loss: 0.0303\n",
            "Iteration: 969; Percent complete: 96.9%; Average loss: 0.0440\n",
            "Iteration: 970; Percent complete: 97.0%; Average loss: 0.0354\n",
            "Iteration: 971; Percent complete: 97.1%; Average loss: 0.0287\n",
            "Iteration: 972; Percent complete: 97.2%; Average loss: 0.0277\n",
            "Iteration: 973; Percent complete: 97.3%; Average loss: 0.0331\n",
            "Iteration: 974; Percent complete: 97.4%; Average loss: 0.0413\n",
            "Iteration: 975; Percent complete: 97.5%; Average loss: 0.0334\n",
            "Iteration: 976; Percent complete: 97.6%; Average loss: 0.0417\n",
            "Iteration: 977; Percent complete: 97.7%; Average loss: 0.0471\n",
            "Iteration: 978; Percent complete: 97.8%; Average loss: 0.0338\n",
            "Iteration: 979; Percent complete: 97.9%; Average loss: 0.0374\n",
            "Iteration: 980; Percent complete: 98.0%; Average loss: 0.0282\n",
            "Iteration: 981; Percent complete: 98.1%; Average loss: 0.0262\n",
            "Iteration: 982; Percent complete: 98.2%; Average loss: 0.0254\n",
            "Iteration: 983; Percent complete: 98.3%; Average loss: 0.0371\n",
            "Iteration: 984; Percent complete: 98.4%; Average loss: 0.0304\n",
            "Iteration: 985; Percent complete: 98.5%; Average loss: 0.0322\n",
            "Iteration: 986; Percent complete: 98.6%; Average loss: 0.0375\n",
            "Iteration: 987; Percent complete: 98.7%; Average loss: 0.0352\n",
            "Iteration: 988; Percent complete: 98.8%; Average loss: 0.0288\n",
            "Iteration: 989; Percent complete: 98.9%; Average loss: 0.0246\n",
            "Iteration: 990; Percent complete: 99.0%; Average loss: 0.0298\n",
            "Iteration: 991; Percent complete: 99.1%; Average loss: 0.0351\n",
            "Iteration: 992; Percent complete: 99.2%; Average loss: 0.0445\n",
            "Iteration: 993; Percent complete: 99.3%; Average loss: 0.0250\n",
            "Iteration: 994; Percent complete: 99.4%; Average loss: 0.0363\n",
            "Iteration: 995; Percent complete: 99.5%; Average loss: 0.0283\n",
            "Iteration: 996; Percent complete: 99.6%; Average loss: 0.0284\n",
            "Iteration: 997; Percent complete: 99.7%; Average loss: 0.0240\n",
            "Iteration: 998; Percent complete: 99.8%; Average loss: 0.0333\n",
            "Iteration: 999; Percent complete: 99.9%; Average loss: 0.0406\n",
            "Iteration: 1000; Percent complete: 100.0%; Average loss: 0.0332\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Evaluation Functions"
      ],
      "metadata": {
        "id": "03W05HyA5Sxh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class GreedySearchDecoder(nn.Module):\n",
        "    def __init__(self, encoder, decoder,device):\n",
        "        super(GreedySearchDecoder, self).__init__()\n",
        "        self.encoder = encoder\n",
        "        self.decoder = decoder\n",
        "        self._device = device\n",
        "\n",
        "    def forward(self, input_seq, input_length, max_length):\n",
        "        # Forward input through encoder model\n",
        "        encoder_outputs, encoder_hidden = self.encoder(input_seq, input_length)\n",
        "        # Prepare encoder's final hidden layer to be first hidden input to the decoder\n",
        "        decoder_hidden = (encoder_hidden[0][:decoder.n_layers], encoder_hidden[1][:decoder.n_layers])\n",
        "        # Initialize decoder input with SOS_token\n",
        "        decoder_input = torch.LongTensor([[SOS_token]])\n",
        "        decoder_input = decoder_input.to(device)\n",
        "        # Initialize tensors to append decoded words to\n",
        "        all_tokens = torch.zeros([0], device=self._device, dtype=torch.long)\n",
        "        all_scores = torch.zeros([0], device=self._device)\n",
        "        # Iteratively decode one word token at a time\n",
        "        for _ in range(max_length):\n",
        "            # Forward pass through decoder\n",
        "            decoder_output, decoder_hidden = self.decoder(decoder_input, decoder_hidden, encoder_outputs)\n",
        "            # Obtain most likely word token and its softmax score\n",
        "            decoder_scores, decoder_input = torch.max(decoder_output, dim=1)\n",
        "            # Record token and score\n",
        "            all_tokens = torch.cat((all_tokens, decoder_input), dim=0)\n",
        "            all_scores = torch.cat((all_scores, decoder_scores), dim=0)\n",
        "            # Prepare current token to be next decoder input (add a dimension)\n",
        "            decoder_input = torch.unsqueeze(decoder_input, 0)\n",
        "        # Return collections of word tokens and scores\n",
        "        return all_tokens, all_scores"
      ],
      "metadata": {
        "id": "3-8tghbd5S5g"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def evaluate(encoder, decoder, searcher, voc, sentence, max_length=MAX_LENGTH):\n",
        "    ### Format input sentence as a batch\n",
        "    # words -> indexes\n",
        "    indexes_batch = [indexesFromSentence(voc, sentence)]\n",
        "    # Create lengths tensor\n",
        "    lengths = torch.tensor([len(indexes) for indexes in indexes_batch])\n",
        "    # Transpose dimensions of batch to match models' expectations\n",
        "    input_batch = torch.LongTensor(indexes_batch).transpose(0, 1)\n",
        "    # Use appropriate device\n",
        "    input_batch = input_batch.to(device)\n",
        "    # lengths = lengths.to(device)\n",
        "    # Decode sentence with searcher\n",
        "    tokens, scores = searcher(input_batch, lengths, max_length)\n",
        "    # indexes -> words\n",
        "    decoded_words = [voc.index2word[token.item()] for token in tokens]\n",
        "    return decoded_words\n",
        "\n",
        "\n",
        "def evaluateInput(encoder, decoder, searcher, voc):\n",
        "    input_sentence = ''\n",
        "    while(1):\n",
        "        try:\n",
        "            # Get input sentence\n",
        "            input_sentence = input('> ')\n",
        "            # Check if it is quit case\n",
        "            if input_sentence == 'q' or input_sentence == 'quit': break\n",
        "            # Normalize sentence\n",
        "            input_sentence = normalizeString(input_sentence)\n",
        "            # Evaluate sentence\n",
        "            output_words = evaluate(encoder, decoder, searcher, voc, input_sentence)\n",
        "            # Format and print response sentence\n",
        "            output_words[:] = [x for x in output_words if not (x == 'EOS' or x == 'PAD')]\n",
        "            print('Bot:', ' '.join(output_words))\n",
        "\n",
        "        except KeyError:\n",
        "            print(\"Error: Encountered unknown word.\")"
      ],
      "metadata": {
        "id": "DI9mvOhQ5-YL"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Manual Evaluation"
      ],
      "metadata": {
        "id": "H_GOPvu_YYwr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Set dropout layers to eval mode\n",
        "encoder.eval()\n",
        "decoder.eval()\n",
        "\n",
        "# Initialize search module\n",
        "searcher = GreedySearchDecoder(encoder, decoder,device)\n",
        "\n",
        "\n",
        "# Begin chatting\n",
        "evaluateInput(encoder, decoder, searcher, voc)"
      ],
      "metadata": {
        "id": "bbU1ngS66Gnq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3908a3ea-d67d-450e-c5cd-5f61f85210d1"
      },
      "execution_count": 19,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "> what is recursion\n",
            "Bot: recursion is a programming technique where a function calls itself in its own definition it allows for solving complex problems by breaking them down into smaller simpler subproblems that are solved recursively recursion can be used to solve problems that exhibit a divide and conquer or topdown approach where a problem is divided into smaller subproblems until a base case is reached recursion can be powerful but should be used with caution to prevent infinite loops or stack overflow errors\n",
            "> Explain the concept of recursion to me\n",
            "Bot: recursion is a programming technique where a function calls itself in its own definition it allows for solving complex problems by breaking them down into smaller simpler subproblems that are solved recursively recursion can be used to solve problems that exhibit a divide and conquer or topdown approach where a problem is divided into smaller subproblems until a base case is reached recursion can be powerful but should be used with caution to prevent infinite loops or stack overflow errors\n",
            "> define a kernel\n",
            "Bot: kernel is the part of os which handles all details of sharing resources and device handlingit can be considered as the core of os which manages the core features of an osits purpose is to handle the communication between software and hardwareits services are used through system calls and a layer of software called shell wraps around the kernel\n",
            "> Can you explain what is a kernel?\n",
            "Bot: kernel is the part of os which handles all details of sharing resources and device handlingit can be considered as the core of os which manages the core features of an osits purpose is to handle the communication between software and hardwareits services are used through system calls and a layer of software called shell wraps around the kernel\n",
            "> explain in brief about data imputation\n",
            "Bot: data imputation is the process of filling in missing or incomplete values in the dataset using statistical techniques such as mean imputation median imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation imputation\n",
            "> how do you explain a priority queue\n",
            "Bot: the priority queue is an ordered collection of elements in which insertions are restricted to one end called the front end and the rear end the rear end is where we insert elements and front end is where we delete elements we can traverse in a circular queue in only one direction ie from front to rear processed another priority of the queue in which the intrinsic position is calculated with the highest end is called the correct priority of the queue the item at the priority or first priority it must be inserted or there are two types of priority queue here smallest priority is not deleted priority first here end is inserted with a elements priority with no jobs should have more than the elements of an array it is a result priority queue here largest insertion is one of the priority queue with every value from the queue of the priority to its largest value with the largest priority\n",
            "> what are the disadvantages array implementations of linked list\n",
            "Bot: the linked list carried represent in data types cant represent the nodes and the fundamental nodes where the list add no element is not allocated it cant be necessary to the list in a linked list it is not within a linked list may be declared some pointer then we need to any size that they can start memory pointer to a node be contiguous pointer delete we need to delete a node in which insertions and deletions are placed at one end the end of the linked list the removal of linked list it is no useful in ubuntu order in which the declaration insertions and deletions can be handled easily and be used to insert the efficiency and reliability of the list as then we can not upon a pointer in a sequence of nodes each structure contains the list in a linked list like store it is allocated to any data points to a list of specific than the same node is called a linked list can be handled easily and efficientlya linked list does not waste any memory space\n",
            "> q\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### BERT Score evaluation"
      ],
      "metadata": {
        "id": "n2WXvkcxkj6z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install bert_score\n"
      ],
      "metadata": {
        "id": "9VpqdBv6kkIC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0fd55bf7-bd8c-45f1-e718-79a1eb6ecd7a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting bert_score\n",
            "  Downloading bert_score-0.3.13-py3-none-any.whl.metadata (15 kB)\n",
            "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from bert_score) (2.5.1+cu121)\n",
            "Requirement already satisfied: pandas>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from bert_score) (2.2.2)\n",
            "Requirement already satisfied: transformers>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from bert_score) (4.46.2)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from bert_score) (1.26.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from bert_score) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.31.1 in /usr/local/lib/python3.10/dist-packages (from bert_score) (4.66.6)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from bert_score) (3.8.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from bert_score) (24.2)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert_score) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert_score) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert_score) (2024.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (3.16.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (4.12.2)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (2024.10.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch>=1.0.0->bert_score) (1.3.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (0.26.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (6.0.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (2024.9.11)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (0.4.5)\n",
            "Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (0.20.3)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (4.55.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (1.4.7)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (11.0.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (3.2.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->bert_score) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->bert_score) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->bert_score) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->bert_score) (2024.8.30)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas>=1.0.1->bert_score) (1.16.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.0.0->bert_score) (3.0.2)\n",
            "Downloading bert_score-0.3.13-py3-none-any.whl (61 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.1/61.1 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: bert_score\n",
            "Successfully installed bert_score-0.3.13\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# check your installation\n",
        "import bert_score\n",
        "print(bert_score.__version__)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K3Xg2cSVkxof",
        "outputId": "2c014171-109c-4c75-9eab-93c80212cb48"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.3.12\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get n random unique questions from questions\n",
        "from bert_score import score\n",
        "questions_evaluation = [q[0] for q in qa_pairs]\n",
        "answers_evaluation = [q[1] for q in qa_pairs]"
      ],
      "metadata": {
        "id": "f7mPge2k681Z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def evaluate(encoder, decoder, searcher, voc, sentence, max_length=MAX_LENGTH):\n",
        "    ### Format input sentence as a batch\n",
        "    # words -> indexes\n",
        "    indexes_batch = [indexesFromSentence(voc, sentence)]\n",
        "    # Create lengths tensor\n",
        "    lengths = torch.tensor([len(indexes) for indexes in indexes_batch])\n",
        "    # Transpose dimensions of batch to match models' expectations\n",
        "    input_batch = torch.LongTensor(indexes_batch).transpose(0, 1)\n",
        "    # Use appropriate device\n",
        "    input_batch = input_batch.to(device)\n",
        "    # lengths = lengths.to(device)\n",
        "    # Decode sentence with searcher\n",
        "    tokens, scores = searcher(input_batch, lengths, max_length)\n",
        "    # indexes -> words\n",
        "    decoded_words = [voc.index2word[token.item()] for token in tokens]\n",
        "    return decoded_words"
      ],
      "metadata": {
        "id": "SVU1-U6rY2qR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Set dropout layers to eval mode\n",
        "encoder.eval()\n",
        "decoder.eval()\n",
        "\n",
        "# Initialize search module\n",
        "searcher = GreedySearchDecoder(encoder, decoder,device)"
      ],
      "metadata": {
        "id": "6vu6Y7XaY2qS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "generated_answers = []\n",
        "for question in questions_evaluation:\n",
        "    question = normalizeString(question)\n",
        "    # Evaluate sentence\n",
        "    output_words = evaluate(encoder, decoder, searcher, voc, question)\n",
        "    # Format and print response sentence\n",
        "    output_words[:] = [x for x in output_words if not (x == 'EOS' or x == 'PAD')]\n",
        "    generated_answers.append(' '.join(output_words))"
      ],
      "metadata": {
        "id": "O3mAVHCHaLMw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Compute the BERT score for each generated answer and the answer\n",
        "eval_df = pd.DataFrame({'question': questions_evaluation, 'answer': answers_evaluation, 'generated_answer': generated_answers})\n",
        "P, R, F1 = score(generated_answers, answers_evaluation, lang='en', verbose=True, rescale_with_baseline=True)"
      ],
      "metadata": {
        "id": "f_ZGp7iuc2Ft",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 377,
          "referenced_widgets": [
            "8d6fb91fb4ae432c805855671f6588ee",
            "0e0bae4bf3604828ab15c5b41c68204c",
            "385e928dbda541f3a2829b5482809cb0",
            "2f22ab3e711748f9a6ca8af2841afb17",
            "fadcda8839f54fe3a865459e7720f14f",
            "d023990f61d44780b8dc117db81802a1",
            "74a3570c1a9f492c963c4301046e9b7d",
            "441a84e9aa0a42d5be229415529a3223",
            "4bedb524641347119d7ba8796b609630",
            "df4067aa7b2f4712b4261c4456140897",
            "74f3a9b0749e46d9938d547ab8e24ada",
            "0ba3e2869d1447f6a16dca4bd13988f6",
            "0bb8bc2bb44b4947b4317391e975667b",
            "ba8470e2b3394afeb6733c3b2713a09a",
            "292550039b264a22a6d5714f0fdaf875",
            "04f17117fc454b6ab7cc9fc873192757",
            "98c84e2f17764a6dac1b25610b98996d",
            "1c1b8aedd3ec44d5ba96fc8a05acef67",
            "8bdb13dabec343488496c7fb715005a0",
            "6970a06ee6624e27b9c1820e91dd4d32",
            "8ff22d3499d8413c9ce6b1ba9525b3d8",
            "879d64ef7ecc41d99e7eed38d7c32e94",
            "8423ae606baf4adf89a7f055863e8273",
            "1d9a2bf0674249b2a75fac7227c62326",
            "3ce9642b5983469c8945d03f402965a4",
            "45e6b592b62d4c7ba726dd2d1b2ed5e7",
            "b29c32d002f24dcaadb29448085fa10b",
            "8aa8edce2fc749d098f771b1631f8eeb",
            "8440418089d347cd8f9652bd82b4f646",
            "07d3ad18fffc46739126d16223bf1f36",
            "dba59af6dd36403cba7c786cb10b5230",
            "aaa2961917ae4670a03eaa8ad3b1eceb",
            "b3bfb60a54d04cb1a1b1a417a9285c99",
            "31c8fff9cc82434e8302583e680be25b",
            "78ad702c68204167a52f5877e10cc083",
            "841afb636a3b4cc7930e48a24bc819be",
            "f48447494d124908963aa06a4fc14bc1",
            "12d3aa1371c64c6ea4b65e6b4acf3f5d",
            "5dacb348757345719236e6ce4cae94d6",
            "33fa547c473f4cd8aa3c419b8ce13eb2",
            "73770ee07714463f937ce66794bcb312",
            "34ccc267f51240dc82a6f6e0251fa001",
            "6c0a2762b27b48c1a1cd5cfc08d07138",
            "ba7cd807f8fa42e598c95082cd1f9baa",
            "3043d07eafdb4da38028c391f78a84f5",
            "dbc45053be0c4b3aa45955c3d8a8584a",
            "3a6eca0e70194b358aa65f9bd88bcb59",
            "25fdf9dc30bc40dea703e73148e58d72",
            "e05b400e72a643b3ad1ce2109c2d39d9",
            "829408ab3d774576934533426f96f54a",
            "5b547a605363495d83f28fb7e3cfeee0",
            "e8c26d57be814c63b0af396a79fd6b35",
            "0c5bb290771245cdbb46a51fa884a43f",
            "4409a422e0044ee89feaea4c8a77c50b",
            "4edc1cf70fde48f19e802453d2f149dc",
            "372a4174aa8646bd8bb7e862ba4478c6",
            "64421d19f2f74899ace180e3f9f637dc",
            "170a382a6d904a84b51cfedf96ef8ad6",
            "76e9d4beb44d44b5afe47e629c8d2bcd",
            "cb9b17ab32dd431498bc37b2c69fe381",
            "811a0af59b1949c9931164b370a96c5c",
            "53a2e763446e4b9b97b6b03efadbb6f1",
            "8781836496a746d58a2f89f5da3e8a34",
            "6177a9d4ad084106bb68a496b350b9c5",
            "30e1d46376694f9b9b2320e63d51629a",
            "eb7d02c498e9417dba6f463433a33538",
            "bdd6ede86666411ea4ce1419c30abfbb",
            "ade4488bd57642dab4675597bacaab60",
            "a55b759e9c834eb6a05d4981e01de108",
            "f30b7ef7c31740c1baddf4b6c9a724b8",
            "6287e9ad00544dd3ae0727c31ffc8349",
            "96c0e0f7cfcb4946bb1cf2a05534a557",
            "de68a7afece7455286466d146175997e",
            "aa0f754bfb2d417b93bd15160f3ba678",
            "39a52cddf11a4261a8deb2891c2e3c46",
            "0e0892d9bf1d4b69aaead621f8378cf0",
            "51375bbdb8364c28abf3f0cfec45df13",
            "efc9007936e74ebfa7a86ef808272cde",
            "60c423dd344f47e9b2bcc34a1cc65278",
            "031dcc01045c4ac0a08c045a0fe1aeee",
            "007d53f6c2a14024b5184e2925923d98",
            "13612dec34fb42babe60b43a3a69d7cc",
            "f2b27f4fdb43401db08cdf5caf2305f4",
            "1933c54824654d8c90d7ef46f50c68c3",
            "ac6f4dd9215543c2840e670f760f1114",
            "3b8bc43ba42b478db8c10440c94b87e9",
            "b1a5ee94af4d4e05b7135c01618a1ddd",
            "b10e7315d8ce4ef8a1e2a4579f5dbd46"
          ]
        },
        "outputId": "5ff6d49b-7671-485b-d6c5-6fde2f016dc5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/25.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "8d6fb91fb4ae432c805855671f6588ee"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/482 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0ba3e2869d1447f6a16dca4bd13988f6"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "vocab.json:   0%|          | 0.00/899k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "8423ae606baf4adf89a7f055863e8273"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "31c8fff9cc82434e8302583e680be25b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "3043d07eafdb4da38028c391f78a84f5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model.safetensors:   0%|          | 0.00/1.42G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "372a4174aa8646bd8bb7e862ba4478c6"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of RobertaModel were not initialized from the model checkpoint at roberta-large and are newly initialized: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "calculating scores...\n",
            "computing bert embedding.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/30 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bdd6ede86666411ea4ce1419c30abfbb"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "computing greedy matching.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/59 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "efc9007936e74ebfa7a86ef808272cde"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "done in 5.97 seconds, 632.73 sentences/sec\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "eval_df['P'] = P\n",
        "eval_df['R'] = R\n",
        "eval_df['F1'] = F1\n",
        "# save to csv\n",
        "save_path = f'{FOLDER_PATH}/evaluation/{model_name}.csv'\n",
        "eval_df.to_csv(save_path, index=False)\n",
        "print(f\"saved to {save_path}\")\n",
        "eval_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        },
        "id": "TmQdjNfkeALx",
        "outputId": "2484f1e0-99ef-4bd5-b78f-b9e5a7cc2628"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "saved to /content/drive/My Drive/Colab Notebooks/nlc/project/evaluation/lstm-general-2-2-1024.csv\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                 question  \\\n",
              "0      how does randomised algorithm work   \n",
              "1    what do you mean by bestfirst search   \n",
              "2             how do you explain a daemon   \n",
              "3              what is phonetic algorithm   \n",
              "4  what do you mean by uniform costsearch   \n",
              "\n",
              "                                              answer  \\\n",
              "0  the algorithm typically uses uniformly random ...   \n",
              "1  bestfirst search is a search algorithm which e...   \n",
              "2  daemon disk and execution monitor is a process...   \n",
              "3  a phonetic algorithm is an algorithm for index...   \n",
              "4  a tree search that finds the lowestcost route ...   \n",
              "\n",
              "                                    generated_answer         P         R  \\\n",
              "0  the algorithm typically uses uniformly random ...  1.000000  1.000000   \n",
              "1  bestfirst search is a search algorithm which e...  1.000000  1.000000   \n",
              "2  daemon is a program that runs in the backgroun...  1.000000  1.000000   \n",
              "3  a phonetic algorithm is an algorithm for index...  0.999999  0.999999   \n",
              "4  a tree search that finds the lowestcost route ...  1.000000  1.000000   \n",
              "\n",
              "         F1  \n",
              "0  1.000000  \n",
              "1  1.000000  \n",
              "2  1.000000  \n",
              "3  0.999999  \n",
              "4  1.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-349d1064-bc4c-4f56-82d7-de80ae3b03c2\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>generated_answer</th>\n",
              "      <th>P</th>\n",
              "      <th>R</th>\n",
              "      <th>F1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>how does randomised algorithm work</td>\n",
              "      <td>the algorithm typically uses uniformly random ...</td>\n",
              "      <td>the algorithm typically uses uniformly random ...</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>what do you mean by bestfirst search</td>\n",
              "      <td>bestfirst search is a search algorithm which e...</td>\n",
              "      <td>bestfirst search is a search algorithm which e...</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>how do you explain a daemon</td>\n",
              "      <td>daemon disk and execution monitor is a process...</td>\n",
              "      <td>daemon is a program that runs in the backgroun...</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>what is phonetic algorithm</td>\n",
              "      <td>a phonetic algorithm is an algorithm for index...</td>\n",
              "      <td>a phonetic algorithm is an algorithm for index...</td>\n",
              "      <td>0.999999</td>\n",
              "      <td>0.999999</td>\n",
              "      <td>0.999999</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>what do you mean by uniform costsearch</td>\n",
              "      <td>a tree search that finds the lowestcost route ...</td>\n",
              "      <td>a tree search that finds the lowestcost route ...</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-349d1064-bc4c-4f56-82d7-de80ae3b03c2')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-349d1064-bc4c-4f56-82d7-de80ae3b03c2 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-349d1064-bc4c-4f56-82d7-de80ae3b03c2');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-d1c5b456-55bd-4c72-8369-8e3299a16c45\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-d1c5b456-55bd-4c72-8369-8e3299a16c45')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-d1c5b456-55bd-4c72-8369-8e3299a16c45 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "eval_df",
              "summary": "{\n  \"name\": \"eval_df\",\n  \"rows\": 3775,\n  \"fields\": [\n    {\n      \"column\": \"question\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3298,\n        \"samples\": [\n          \"define bidirectional search\",\n          \"what do you mean by divide and conquer algorithm\",\n          \"what is computational intelligence\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"answer\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1815,\n        \"samples\": [\n          \"a time sharing method must be implemented to allow multiple processes to have an access to the system this will involve the preemption of processes that do not give up cpu on their own ie more than one process may be executing kernel code simultaneouslythe amount of resources that a process can use and the operations that it may perform must be limited the system resources and the processes must be protected from each otherkernel must be designed to prevent deadlocks between the various processes ie cyclic waiting or hold and waiting must not occureffective memory management techniques must be used to better utilize the limited resources\",\n          \"maintaining a set of connected components of a graphmaintain list of duplicate copies of web pages constructing a minimum spanning tree for a graph define an equivalence relation\",\n          \"cache memory is a small highspeed memory that sits between the cpu and main memory in a computer system it stores frequently accessed data and instructions to reduce the cpus access time to main memory thereby improving system performance cache memory is an important component of the memory hierarchy and is typically organized into multiple levels such as l l and l caches\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"generated_answer\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1707,\n        \"samples\": [\n          \"a lineartime online algorithm for constructing suffix trees\",\n          \"stochastic search is a hyperparameter method that sequentially hyperparameter points from the distance between one space and smooth information commonly used in computer graphics curve fitting and data recognition\",\n          \"java is a compiled staticallytyped language used to solve combinational and java javascript javascript divides a main environment to access code while thoroughness and override memory while responsibilities\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"P\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 308,\n        \"samples\": [\n          0.933729350566864,\n          -0.19373948872089386,\n          0.9226308465003967\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"R\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 307,\n        \"samples\": [\n          0.8130138516426086,\n          0.9666504859924316,\n          0.9094600081443787\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"F1\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 304,\n        \"samples\": [\n          0.8988099098205566,\n          0.11170830577611923,\n          0.7924590706825256\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# average BERT Score\n",
        "print(f'Avg. P: {P.mean():.4f}, Avg. R: {R.mean():.4f}, Avg. F1: {F1.mean():.4f}')\n",
        "# minimum BERT Score\n",
        "print(f'Min. P: {P.min():.4f}, Min. R: {R.min():.4f}, Min. F1: {F1.min():.4f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JWQj7JcNlAGp",
        "outputId": "81b56841-2db4-44bc-d604-4064c2d20c26"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. P: 0.9521, Avg. R: 0.9543, Avg. F1: 0.9531\n",
            "Min. P: -0.6662, Min. R: -0.1103, Min. F1: -0.2307\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Display the questions and generated answers who have negative f1\n",
        "poor_eval_df = eval_df[eval_df['F1'] < 0]\n",
        "# sort by f1 in descending\n",
        "poor_eval_df = poor_eval_df.sort_values(by='F1', ascending=True)\n",
        "poor_eval_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 394
        },
        "id": "8WvWGHw7mELg",
        "outputId": "31d7cac1-ce1e-44fa-f52b-53faa2da14c7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                               question  \\\n",
              "3070             explain in brief about data imputation   \n",
              "1691  what are the disadvantages array implementatio...   \n",
              "2636                   what are the states of a process   \n",
              "57                  how do you explain topological sort   \n",
              "1008                           what is topological sort   \n",
              "2221                            define topological sort   \n",
              "399                what do you mean by topological sort   \n",
              "224                what do you mean by a priority queue   \n",
              "1198                how do you explain a priority queue   \n",
              "1493  what are the different types of scheduling alg...   \n",
              "795                      define ancestor and descendant   \n",
              "\n",
              "                                                 answer  \\\n",
              "3070  data imputation is the process of filling in m...   \n",
              "1691  the no of nodes needed cant be predicted when ...   \n",
              "2636             new runnin waitingready and terminated   \n",
              "57    finds linear order of nodes eg jobs based on t...   \n",
              "1008  finds linear order of nodes eg jobs based on t...   \n",
              "2221  finds linear order of nodes eg jobs based on t...   \n",
              "399   finds linear order of nodes eg jobs based on t...   \n",
              "224   the priority queue is a data structure in whic...   \n",
              "1198  the priority queue is a data structure in whic...   \n",
              "1493  the scheduling algorithms decide which process...   \n",
              "795   an expression tree is a tree which is build fr...   \n",
              "\n",
              "                                       generated_answer         P         R  \\\n",
              "3070  data imputation is the process of filling in m... -0.666181  0.298758   \n",
              "1691  the no of nodes needed cant be declared a list... -0.311747 -0.070371   \n",
              "2636             new runnin waitingready and terminated -0.258682 -0.100382   \n",
              "57    finds linear order of nodes eg jobs based on t... -0.193739  0.029004   \n",
              "1008  finds linear order of nodes eg jobs based on t... -0.193739  0.029004   \n",
              "2221  finds linear order of nodes eg jobs based on t... -0.193739  0.029004   \n",
              "399   finds linear order of nodes eg jobs based on t... -0.193739  0.029004   \n",
              "224   the priority queue is a data structure in whic... -0.132608  0.002581   \n",
              "1198  the priority queue is a data structure in whic... -0.117299  0.040921   \n",
              "1493  the scheduling algorithms lifo insertion under... -0.089561  0.064208   \n",
              "795   if there is a path from node n to n then n is ...  0.042239 -0.065550   \n",
              "\n",
              "            F1  \n",
              "3070 -0.230679  \n",
              "1691 -0.192183  \n",
              "2636 -0.178927  \n",
              "57   -0.083156  \n",
              "1008 -0.083156  \n",
              "2221 -0.083156  \n",
              "399  -0.083156  \n",
              "224  -0.064217  \n",
              "1198 -0.037775  \n",
              "1493 -0.012226  \n",
              "795  -0.010603  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-20d5f92f-5362-4f24-b598-27a490102634\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>generated_answer</th>\n",
              "      <th>P</th>\n",
              "      <th>R</th>\n",
              "      <th>F1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>3070</th>\n",
              "      <td>explain in brief about data imputation</td>\n",
              "      <td>data imputation is the process of filling in m...</td>\n",
              "      <td>data imputation is the process of filling in m...</td>\n",
              "      <td>-0.666181</td>\n",
              "      <td>0.298758</td>\n",
              "      <td>-0.230679</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1691</th>\n",
              "      <td>what are the disadvantages array implementatio...</td>\n",
              "      <td>the no of nodes needed cant be predicted when ...</td>\n",
              "      <td>the no of nodes needed cant be declared a list...</td>\n",
              "      <td>-0.311747</td>\n",
              "      <td>-0.070371</td>\n",
              "      <td>-0.192183</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2636</th>\n",
              "      <td>what are the states of a process</td>\n",
              "      <td>new runnin waitingready and terminated</td>\n",
              "      <td>new runnin waitingready and terminated</td>\n",
              "      <td>-0.258682</td>\n",
              "      <td>-0.100382</td>\n",
              "      <td>-0.178927</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>how do you explain topological sort</td>\n",
              "      <td>finds linear order of nodes eg jobs based on t...</td>\n",
              "      <td>finds linear order of nodes eg jobs based on t...</td>\n",
              "      <td>-0.193739</td>\n",
              "      <td>0.029004</td>\n",
              "      <td>-0.083156</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1008</th>\n",
              "      <td>what is topological sort</td>\n",
              "      <td>finds linear order of nodes eg jobs based on t...</td>\n",
              "      <td>finds linear order of nodes eg jobs based on t...</td>\n",
              "      <td>-0.193739</td>\n",
              "      <td>0.029004</td>\n",
              "      <td>-0.083156</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2221</th>\n",
              "      <td>define topological sort</td>\n",
              "      <td>finds linear order of nodes eg jobs based on t...</td>\n",
              "      <td>finds linear order of nodes eg jobs based on t...</td>\n",
              "      <td>-0.193739</td>\n",
              "      <td>0.029004</td>\n",
              "      <td>-0.083156</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>399</th>\n",
              "      <td>what do you mean by topological sort</td>\n",
              "      <td>finds linear order of nodes eg jobs based on t...</td>\n",
              "      <td>finds linear order of nodes eg jobs based on t...</td>\n",
              "      <td>-0.193739</td>\n",
              "      <td>0.029004</td>\n",
              "      <td>-0.083156</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>224</th>\n",
              "      <td>what do you mean by a priority queue</td>\n",
              "      <td>the priority queue is a data structure in whic...</td>\n",
              "      <td>the priority queue is a data structure in whic...</td>\n",
              "      <td>-0.132608</td>\n",
              "      <td>0.002581</td>\n",
              "      <td>-0.064217</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1198</th>\n",
              "      <td>how do you explain a priority queue</td>\n",
              "      <td>the priority queue is a data structure in whic...</td>\n",
              "      <td>the priority queue is a data structure in whic...</td>\n",
              "      <td>-0.117299</td>\n",
              "      <td>0.040921</td>\n",
              "      <td>-0.037775</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1493</th>\n",
              "      <td>what are the different types of scheduling alg...</td>\n",
              "      <td>the scheduling algorithms decide which process...</td>\n",
              "      <td>the scheduling algorithms lifo insertion under...</td>\n",
              "      <td>-0.089561</td>\n",
              "      <td>0.064208</td>\n",
              "      <td>-0.012226</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>795</th>\n",
              "      <td>define ancestor and descendant</td>\n",
              "      <td>an expression tree is a tree which is build fr...</td>\n",
              "      <td>if there is a path from node n to n then n is ...</td>\n",
              "      <td>0.042239</td>\n",
              "      <td>-0.065550</td>\n",
              "      <td>-0.010603</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-20d5f92f-5362-4f24-b598-27a490102634')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-20d5f92f-5362-4f24-b598-27a490102634 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-20d5f92f-5362-4f24-b598-27a490102634');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-054c4f74-671d-41c4-a66b-3af9bc25d643\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-054c4f74-671d-41c4-a66b-3af9bc25d643')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-054c4f74-671d-41c4-a66b-3af9bc25d643 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_6de370de-8aea-4127-b527-218b31448e6d\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('poor_eval_df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_6de370de-8aea-4127-b527-218b31448e6d button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('poor_eval_df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "poor_eval_df",
              "summary": "{\n  \"name\": \"poor_eval_df\",\n  \"rows\": 11,\n  \"fields\": [\n    {\n      \"column\": \"question\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 11,\n        \"samples\": [\n          \"define topological sort\",\n          \"explain in brief about data imputation\",\n          \"what are the different types of scheduling algorithms\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"answer\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 7,\n        \"samples\": [\n          \"data imputation is the process of filling in missing or incomplete values in the dataset using statistical techniques such as mean imputation median imputation mode imputation or predictive imputation enabling more robust analysis and modeling of incomplete data\",\n          \"the no of nodes needed cant be predicted when the program is writtenthe no of nodes declared must remain allocated throughout its execution\",\n          \"the scheduling algorithms decide which processes in the ready queue are to be allocated to the cpu for execution scheduling algorithms can be broadly classified on the basis of preemptive algorithmsround robin schedulingshortest job first scheduling can be bothpriority scheduling can be bothnonpreemptive\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"generated_answer\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 7,\n        \"samples\": [\n          \"data imputation is the process of filling in missing or incomplete values in the dataset using statistical techniques such as mean imputation median imputation relationship or singular representation\",\n          \"the no of nodes needed cant be declared a list is allocated a list to its last ie nodes by no linked list is only a linked list calls it is needed to any new node cannot be declared a linked number with any first order where its number is equal to its pointer than that is equal to the element of the list\",\n          \"the scheduling algorithms lifo insertion under associated in a unique is called order with any empty cell with b first come first served a situation is the simplest portion of a time process allows that allows about when one to one required at a time\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"P\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          -0.31174734234809875,\n          -0.11729857325553894,\n          -0.666181206703186\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"R\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          -0.0703706294298172,\n          0.04092060402035713,\n          0.29875773191452026\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"F1\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          -0.19218333065509796,\n          -0.03777455538511276,\n          -0.23067869246006012\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Visualisation\n",
        "Visualise the distribution of positive and negative BERT score answers"
      ],
      "metadata": {
        "id": "0KnlYnMHgvqf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install plotly-express"
      ],
      "metadata": {
        "id": "Ngvsei3hgxxK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import plotly.express as px"
      ],
      "metadata": {
        "id": "mkXkqwIeg2DZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# plot histogram of the f1 using plotly express\n",
        "fig = px.histogram(eval_df[eval_df['F1'] < 0], x='F1', nbins=20, title='Histogram Plot of BERT (F1) Score of Generated Answer vs Answer')\n",
        "fig.update_layout(xaxis_title='F1 Score', yaxis_title='Count')\n",
        "fig.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "yMdeWNw1ma-h",
        "outputId": "2a7359b6-a498-4f18-a75a-6c51d8d6cf8b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script charset=\"utf-8\" src=\"https://cdn.plot.ly/plotly-2.35.2.min.js\"></script>                <div id=\"3230ecc6-03e4-4c7d-8f9b-7978489eb5a6\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"3230ecc6-03e4-4c7d-8f9b-7978489eb5a6\")) {                    Plotly.newPlot(                        \"3230ecc6-03e4-4c7d-8f9b-7978489eb5a6\",                        [{\"alignmentgroup\":\"True\",\"bingroup\":\"x\",\"hovertemplate\":\"F1=%{x}\\u003cbr\\u003ecount=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"\",\"nbinsx\":20,\"offsetgroup\":\"\",\"orientation\":\"v\",\"showlegend\":false,\"x\":[-0.053061172,-0.06062237,-0.050276842,-0.06062237,-0.0033087744,-0.17406647,-0.06937867,-0.08315587,-0.08315587,-0.020743322,-0.18838437,-0.044495583,-0.021777732,-0.010953671,-0.04271953,-0.015543364,-0.1768794,-0.017508712,-0.121043436,-0.16338755,-0.0629543,-0.0058688456,-0.08598117,-0.031177845,-0.13079071,-0.05036937,-0.05358244,-0.050281435,-0.072663784,-0.015335352,-0.097360045,-0.11815952,-0.009686173,-0.09665866,-0.117860034,-0.072751366,-0.07958681,-0.043970432,-0.023675624,-0.1332367,-0.13513814,-0.057335492,-0.012743496,-0.06062237,-0.10374486,-0.0591066,-0.041727852,-0.06062237,-0.14756131,-0.10236188,-0.023968749,-0.08315587,-0.25497058,-0.051814158,-0.032438632,-0.010602982,-0.08315587,-0.011742989,-0.017685998,-0.063466385,-0.109716825,-0.0069661196,-0.01620519,-0.042769328,-0.011262688,-0.010279485,-0.014552039,-0.045516927,-0.005987861,-0.0564095,-0.017165791,-0.04855624,-0.16137101,-0.027183935,-0.066838376,-0.14351478,-0.044359263,-0.20547916,-0.105846174,-0.08381699,-0.041996256,-0.021413622,-0.058982287,-0.02231807,-0.012637194,-0.009222471,-0.009619779,-0.041830268,-0.15263978,-0.05505795,-0.005252578,-0.005531576,-0.050773744,-0.00025639558,-0.16480973,-0.0021924295,-0.00346911,-0.08092035],\"xaxis\":\"x\",\"yaxis\":\"y\",\"type\":\"histogram\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"F1 Score\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"Count\"}},\"legend\":{\"tracegroupgap\":0},\"title\":{\"text\":\"Histogram Plot of BERT (F1) Score of Generated Answer vs Answer\"},\"barmode\":\"relative\"},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('3230ecc6-03e4-4c7d-8f9b-7978489eb5a6');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Hyperparameter Evaluation\n",
        "\n",
        "Compare the average BERT score across the different hyperparameters"
      ],
      "metadata": {
        "id": "Q5PcvSB7KzLE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Compare the BERT score\n",
        "# load the evaluation csv files\n",
        "import pandas as pd\n",
        "evaluation_csv_files = ['lstm-dot-2-2-1024.csv', 'lstm-dot-4-4-1024.csv', 'lstm-general-2-2-1024.csv', 'lstm-general-4-4-1024.csv']\n",
        "highest_f1 = 0\n",
        "for eval_file in evaluation_csv_files:\n",
        "    eval_df = pd.read_csv(f'{FOLDER_PATH}/evaluation/{eval_file}')\n",
        "    print(f'file: {eval_file}\\t Avg. P: {eval_df[\"P\"].mean():.4f}, Avg. R: {eval_df[\"R\"].mean():.4f}, Avg. F1: {eval_df[\"F1\"].mean():.4f}')\n",
        "    if eval_df['F1'].mean() > highest_f1:\n",
        "        highest_f1 = eval_df['F1'].mean()\n",
        "        highest_f1_file = eval_file\n",
        "print(f'{highest_f1_file} has the highest avg bert score: {highest_f1}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yxgBb3hULPck",
        "outputId": "59c7abce-9836-42f6-b958-ee4545ef2d01"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "file: lstm-dot-2-2-1024.csv\t Avg. P: 0.9494, Avg. R: 0.9537, Avg. F1: 0.9515\n",
            "file: lstm-dot-4-4-1024.csv\t Avg. P: 0.7872, Avg. R: 0.7901, Avg. F1: 0.7884\n",
            "file: lstm-general-2-2-1024.csv\t Avg. P: 0.9521, Avg. R: 0.9543, Avg. F1: 0.9531\n",
            "file: lstm-general-4-4-1024.csv\t Avg. P: 0.8036, Avg. R: 0.8068, Avg. F1: 0.8049\n",
            "lstm-general-2-2-1024.csv has the highest avg bert score: 0.9531026439086092\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "LSTM with 2 encoder and 2 decoder layers, 1024 hidden size using dot attention gives the highest BERT Score"
      ],
      "metadata": {
        "id": "ho_AMnofLPmb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Telegram Implementation"
      ],
      "metadata": {
        "id": "icPY3cbo3ksU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install python-telegram-bot --upgrade\n",
        "!pip install nest_asyncio"
      ],
      "metadata": {
        "id": "_kqmgHsursMX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "51e40151-7c03-486a-981d-b7b6f16960f4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting python-telegram-bot\n",
            "  Downloading python_telegram_bot-21.7-py3-none-any.whl.metadata (17 kB)\n",
            "Requirement already satisfied: httpx~=0.27 in /usr/local/lib/python3.10/dist-packages (from python-telegram-bot) (0.27.2)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (3.7.1)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (2024.8.30)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (1.0.7)\n",
            "Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (3.10)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (1.3.1)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx~=0.27->python-telegram-bot) (0.14.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx~=0.27->python-telegram-bot) (1.2.2)\n",
            "Downloading python_telegram_bot-21.7-py3-none-any.whl (654 kB)\n",
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/654.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━\u001b[0m \u001b[32m634.9/654.9 kB\u001b[0m \u001b[31m20.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m654.9/654.9 kB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: python-telegram-bot\n",
            "Successfully installed python-telegram-bot-21.7\n",
            "Requirement already satisfied: nest_asyncio in /usr/local/lib/python3.10/dist-packages (1.6.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import logging\n",
        "import asyncio\n",
        "import nest_asyncio\n",
        "from telegram import Update\n",
        "from telegram.ext import Application, MessageHandler, filters, ContextTypes\n",
        "\n",
        "# Apply the nest_asyncio patch for Jupyter environments\n",
        "nest_asyncio.apply()\n",
        "\n",
        "# Configure logging\n",
        "logging.basicConfig(\n",
        "    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',\n",
        "    level=logging.INFO\n",
        ")\n",
        "\n",
        "# Define the function to process user input and generate responses using your model\n",
        "async def respond(update: Update, context: ContextTypes.DEFAULT_TYPE):\n",
        "    user_message = update.message.text\n",
        "    # Preprocess user message if necessary\n",
        "    input_sentence = user_message  # Or use preprocess_sentence(user_message)\n",
        "\n",
        "    # Generate a response using your evaluate function\n",
        "    response = evaluate(encoder, decoder, searcher, input_sentence)\n",
        "\n",
        "    # Send the response back to the user\n",
        "    await update.message.reply_text(response)\n",
        "\n",
        "# Main function to set up and run the bot\n",
        "async def main():\n",
        "    # Replace 'YOUR_TOKEN' with your bot's API token\n",
        "    application = Application.builder().token('Replace token here').build()\n",
        "\n",
        "    # Add handler for text messages\n",
        "    application.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, respond))\n",
        "\n",
        "    # Initialize, start, and poll the bot\n",
        "    await application.initialize()\n",
        "    await application.start()\n",
        "    await application.updater.start_polling()\n",
        "    # Keep the bot running\n",
        "    await asyncio.Event().wait()\n",
        "\n",
        "# Run the bot using asyncio\n",
        "asyncio.run(main())"
      ],
      "metadata": {
        "id": "ARGKGxLa3qmT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 367
        },
        "outputId": "1a5038ac-b2be-48ba-9970-bc9c5d13e416"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-37-01cd50b9cb5e>\u001b[0m in \u001b[0;36m<cell line: 44>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     42\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     43\u001b[0m \u001b[0;31m# Run the bot using asyncio\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 44\u001b[0;31m \u001b[0masyncio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/nest_asyncio.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(main, debug)\u001b[0m\n\u001b[1;32m     28\u001b[0m         \u001b[0mtask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0masyncio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_future\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmain\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 30\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mloop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_until_complete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtask\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     31\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     32\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mtask\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/nest_asyncio.py\u001b[0m in \u001b[0;36mrun_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m     90\u001b[0m                 \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_log_destroy_pending\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m             \u001b[0;32mwhile\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 92\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_once\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     93\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stopping\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     94\u001b[0m                     \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/nest_asyncio.py\u001b[0m in \u001b[0;36m_run_once\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    113\u001b[0m                 scheduled[0]._when - self.time(), 0), 86400) if scheduled\n\u001b[1;32m    114\u001b[0m             else None)\n\u001b[0;32m--> 115\u001b[0;31m         \u001b[0mevent_list\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_selector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mselect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    116\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_process_events\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mevent_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    117\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.10/selectors.py\u001b[0m in \u001b[0;36mselect\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    467\u001b[0m             \u001b[0mready\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    468\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 469\u001b[0;31m                 \u001b[0mfd_event_list\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_selector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpoll\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_ev\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    470\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mInterruptedError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    471\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mready\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "T4",
      "collapsed_sections": [
        "aO1Gqpuvtmud",
        "G3ow6JgothNy",
        "-NASuplC3w-N",
        "JTZIHoO-41rf",
        "SFjeoyzLfRmz",
        "03W05HyA5Sxh"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.5"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "8d6fb91fb4ae432c805855671f6588ee": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0e0bae4bf3604828ab15c5b41c68204c",
              "IPY_MODEL_385e928dbda541f3a2829b5482809cb0",
              "IPY_MODEL_2f22ab3e711748f9a6ca8af2841afb17"
            ],
            "layout": "IPY_MODEL_fadcda8839f54fe3a865459e7720f14f"
          }
        },
        "0e0bae4bf3604828ab15c5b41c68204c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d023990f61d44780b8dc117db81802a1",
            "placeholder": "​",
            "style": "IPY_MODEL_74a3570c1a9f492c963c4301046e9b7d",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "385e928dbda541f3a2829b5482809cb0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_441a84e9aa0a42d5be229415529a3223",
            "max": 25,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_4bedb524641347119d7ba8796b609630",
            "value": 25
          }
        },
        "2f22ab3e711748f9a6ca8af2841afb17": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_df4067aa7b2f4712b4261c4456140897",
            "placeholder": "​",
            "style": "IPY_MODEL_74f3a9b0749e46d9938d547ab8e24ada",
            "value": " 25.0/25.0 [00:00&lt;00:00, 1.95kB/s]"
          }
        },
        "fadcda8839f54fe3a865459e7720f14f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d023990f61d44780b8dc117db81802a1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "74a3570c1a9f492c963c4301046e9b7d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "441a84e9aa0a42d5be229415529a3223": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4bedb524641347119d7ba8796b609630": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "df4067aa7b2f4712b4261c4456140897": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "74f3a9b0749e46d9938d547ab8e24ada": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0ba3e2869d1447f6a16dca4bd13988f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0bb8bc2bb44b4947b4317391e975667b",
              "IPY_MODEL_ba8470e2b3394afeb6733c3b2713a09a",
              "IPY_MODEL_292550039b264a22a6d5714f0fdaf875"
            ],
            "layout": "IPY_MODEL_04f17117fc454b6ab7cc9fc873192757"
          }
        },
        "0bb8bc2bb44b4947b4317391e975667b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_98c84e2f17764a6dac1b25610b98996d",
            "placeholder": "​",
            "style": "IPY_MODEL_1c1b8aedd3ec44d5ba96fc8a05acef67",
            "value": "config.json: 100%"
          }
        },
        "ba8470e2b3394afeb6733c3b2713a09a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8bdb13dabec343488496c7fb715005a0",
            "max": 482,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6970a06ee6624e27b9c1820e91dd4d32",
            "value": 482
          }
        },
        "292550039b264a22a6d5714f0fdaf875": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8ff22d3499d8413c9ce6b1ba9525b3d8",
            "placeholder": "​",
            "style": "IPY_MODEL_879d64ef7ecc41d99e7eed38d7c32e94",
            "value": " 482/482 [00:00&lt;00:00, 35.8kB/s]"
          }
        },
        "04f17117fc454b6ab7cc9fc873192757": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "98c84e2f17764a6dac1b25610b98996d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1c1b8aedd3ec44d5ba96fc8a05acef67": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8bdb13dabec343488496c7fb715005a0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6970a06ee6624e27b9c1820e91dd4d32": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8ff22d3499d8413c9ce6b1ba9525b3d8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "879d64ef7ecc41d99e7eed38d7c32e94": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8423ae606baf4adf89a7f055863e8273": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1d9a2bf0674249b2a75fac7227c62326",
              "IPY_MODEL_3ce9642b5983469c8945d03f402965a4",
              "IPY_MODEL_45e6b592b62d4c7ba726dd2d1b2ed5e7"
            ],
            "layout": "IPY_MODEL_b29c32d002f24dcaadb29448085fa10b"
          }
        },
        "1d9a2bf0674249b2a75fac7227c62326": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8aa8edce2fc749d098f771b1631f8eeb",
            "placeholder": "​",
            "style": "IPY_MODEL_8440418089d347cd8f9652bd82b4f646",
            "value": "vocab.json: 100%"
          }
        },
        "3ce9642b5983469c8945d03f402965a4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_07d3ad18fffc46739126d16223bf1f36",
            "max": 898823,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_dba59af6dd36403cba7c786cb10b5230",
            "value": 898823
          }
        },
        "45e6b592b62d4c7ba726dd2d1b2ed5e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_aaa2961917ae4670a03eaa8ad3b1eceb",
            "placeholder": "​",
            "style": "IPY_MODEL_b3bfb60a54d04cb1a1b1a417a9285c99",
            "value": " 899k/899k [00:00&lt;00:00, 6.87MB/s]"
          }
        },
        "b29c32d002f24dcaadb29448085fa10b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8aa8edce2fc749d098f771b1631f8eeb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8440418089d347cd8f9652bd82b4f646": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "07d3ad18fffc46739126d16223bf1f36": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dba59af6dd36403cba7c786cb10b5230": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "aaa2961917ae4670a03eaa8ad3b1eceb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b3bfb60a54d04cb1a1b1a417a9285c99": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "31c8fff9cc82434e8302583e680be25b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_78ad702c68204167a52f5877e10cc083",
              "IPY_MODEL_841afb636a3b4cc7930e48a24bc819be",
              "IPY_MODEL_f48447494d124908963aa06a4fc14bc1"
            ],
            "layout": "IPY_MODEL_12d3aa1371c64c6ea4b65e6b4acf3f5d"
          }
        },
        "78ad702c68204167a52f5877e10cc083": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5dacb348757345719236e6ce4cae94d6",
            "placeholder": "​",
            "style": "IPY_MODEL_33fa547c473f4cd8aa3c419b8ce13eb2",
            "value": "merges.txt: 100%"
          }
        },
        "841afb636a3b4cc7930e48a24bc819be": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_73770ee07714463f937ce66794bcb312",
            "max": 456318,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_34ccc267f51240dc82a6f6e0251fa001",
            "value": 456318
          }
        },
        "f48447494d124908963aa06a4fc14bc1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6c0a2762b27b48c1a1cd5cfc08d07138",
            "placeholder": "​",
            "style": "IPY_MODEL_ba7cd807f8fa42e598c95082cd1f9baa",
            "value": " 456k/456k [00:00&lt;00:00, 3.49MB/s]"
          }
        },
        "12d3aa1371c64c6ea4b65e6b4acf3f5d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5dacb348757345719236e6ce4cae94d6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "33fa547c473f4cd8aa3c419b8ce13eb2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "73770ee07714463f937ce66794bcb312": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "34ccc267f51240dc82a6f6e0251fa001": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6c0a2762b27b48c1a1cd5cfc08d07138": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ba7cd807f8fa42e598c95082cd1f9baa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3043d07eafdb4da38028c391f78a84f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_dbc45053be0c4b3aa45955c3d8a8584a",
              "IPY_MODEL_3a6eca0e70194b358aa65f9bd88bcb59",
              "IPY_MODEL_25fdf9dc30bc40dea703e73148e58d72"
            ],
            "layout": "IPY_MODEL_e05b400e72a643b3ad1ce2109c2d39d9"
          }
        },
        "dbc45053be0c4b3aa45955c3d8a8584a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_829408ab3d774576934533426f96f54a",
            "placeholder": "​",
            "style": "IPY_MODEL_5b547a605363495d83f28fb7e3cfeee0",
            "value": "tokenizer.json: 100%"
          }
        },
        "3a6eca0e70194b358aa65f9bd88bcb59": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e8c26d57be814c63b0af396a79fd6b35",
            "max": 1355863,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0c5bb290771245cdbb46a51fa884a43f",
            "value": 1355863
          }
        },
        "25fdf9dc30bc40dea703e73148e58d72": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4409a422e0044ee89feaea4c8a77c50b",
            "placeholder": "​",
            "style": "IPY_MODEL_4edc1cf70fde48f19e802453d2f149dc",
            "value": " 1.36M/1.36M [00:00&lt;00:00, 7.70MB/s]"
          }
        },
        "e05b400e72a643b3ad1ce2109c2d39d9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "829408ab3d774576934533426f96f54a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5b547a605363495d83f28fb7e3cfeee0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e8c26d57be814c63b0af396a79fd6b35": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0c5bb290771245cdbb46a51fa884a43f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4409a422e0044ee89feaea4c8a77c50b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4edc1cf70fde48f19e802453d2f149dc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "372a4174aa8646bd8bb7e862ba4478c6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_64421d19f2f74899ace180e3f9f637dc",
              "IPY_MODEL_170a382a6d904a84b51cfedf96ef8ad6",
              "IPY_MODEL_76e9d4beb44d44b5afe47e629c8d2bcd"
            ],
            "layout": "IPY_MODEL_cb9b17ab32dd431498bc37b2c69fe381"
          }
        },
        "64421d19f2f74899ace180e3f9f637dc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_811a0af59b1949c9931164b370a96c5c",
            "placeholder": "​",
            "style": "IPY_MODEL_53a2e763446e4b9b97b6b03efadbb6f1",
            "value": "model.safetensors: 100%"
          }
        },
        "170a382a6d904a84b51cfedf96ef8ad6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8781836496a746d58a2f89f5da3e8a34",
            "max": 1421700479,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6177a9d4ad084106bb68a496b350b9c5",
            "value": 1421700479
          }
        },
        "76e9d4beb44d44b5afe47e629c8d2bcd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_30e1d46376694f9b9b2320e63d51629a",
            "placeholder": "​",
            "style": "IPY_MODEL_eb7d02c498e9417dba6f463433a33538",
            "value": " 1.42G/1.42G [00:06&lt;00:00, 236MB/s]"
          }
        },
        "cb9b17ab32dd431498bc37b2c69fe381": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "811a0af59b1949c9931164b370a96c5c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "53a2e763446e4b9b97b6b03efadbb6f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8781836496a746d58a2f89f5da3e8a34": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6177a9d4ad084106bb68a496b350b9c5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "30e1d46376694f9b9b2320e63d51629a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "eb7d02c498e9417dba6f463433a33538": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bdd6ede86666411ea4ce1419c30abfbb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ade4488bd57642dab4675597bacaab60",
              "IPY_MODEL_a55b759e9c834eb6a05d4981e01de108",
              "IPY_MODEL_f30b7ef7c31740c1baddf4b6c9a724b8"
            ],
            "layout": "IPY_MODEL_6287e9ad00544dd3ae0727c31ffc8349"
          }
        },
        "ade4488bd57642dab4675597bacaab60": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_96c0e0f7cfcb4946bb1cf2a05534a557",
            "placeholder": "​",
            "style": "IPY_MODEL_de68a7afece7455286466d146175997e",
            "value": "100%"
          }
        },
        "a55b759e9c834eb6a05d4981e01de108": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_aa0f754bfb2d417b93bd15160f3ba678",
            "max": 30,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_39a52cddf11a4261a8deb2891c2e3c46",
            "value": 30
          }
        },
        "f30b7ef7c31740c1baddf4b6c9a724b8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0e0892d9bf1d4b69aaead621f8378cf0",
            "placeholder": "​",
            "style": "IPY_MODEL_51375bbdb8364c28abf3f0cfec45df13",
            "value": " 30/30 [00:04&lt;00:00, 12.04it/s]"
          }
        },
        "6287e9ad00544dd3ae0727c31ffc8349": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "96c0e0f7cfcb4946bb1cf2a05534a557": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "de68a7afece7455286466d146175997e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "aa0f754bfb2d417b93bd15160f3ba678": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "39a52cddf11a4261a8deb2891c2e3c46": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0e0892d9bf1d4b69aaead621f8378cf0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "51375bbdb8364c28abf3f0cfec45df13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "efc9007936e74ebfa7a86ef808272cde": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_60c423dd344f47e9b2bcc34a1cc65278",
              "IPY_MODEL_031dcc01045c4ac0a08c045a0fe1aeee",
              "IPY_MODEL_007d53f6c2a14024b5184e2925923d98"
            ],
            "layout": "IPY_MODEL_13612dec34fb42babe60b43a3a69d7cc"
          }
        },
        "60c423dd344f47e9b2bcc34a1cc65278": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f2b27f4fdb43401db08cdf5caf2305f4",
            "placeholder": "​",
            "style": "IPY_MODEL_1933c54824654d8c90d7ef46f50c68c3",
            "value": "100%"
          }
        },
        "031dcc01045c4ac0a08c045a0fe1aeee": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ac6f4dd9215543c2840e670f760f1114",
            "max": 59,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3b8bc43ba42b478db8c10440c94b87e9",
            "value": 59
          }
        },
        "007d53f6c2a14024b5184e2925923d98": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b1a5ee94af4d4e05b7135c01618a1ddd",
            "placeholder": "​",
            "style": "IPY_MODEL_b10e7315d8ce4ef8a1e2a4579f5dbd46",
            "value": " 59/59 [00:00&lt;00:00, 61.83it/s]"
          }
        },
        "13612dec34fb42babe60b43a3a69d7cc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f2b27f4fdb43401db08cdf5caf2305f4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1933c54824654d8c90d7ef46f50c68c3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ac6f4dd9215543c2840e670f760f1114": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3b8bc43ba42b478db8c10440c94b87e9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b1a5ee94af4d4e05b7135c01618a1ddd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b10e7315d8ce4ef8a1e2a4579f5dbd46": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}