DistilRoberta-LSTM_training.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**CS Interview QA Chatbot Usage**\n",
        "\n",
        "This workbook contains the code for training and running the a DistilRoberta-LSTM model that answers question about CS interview questions.\n",
        "\n",
        "Please ensure that you have ran the preprocessing.ipynb notebook before running this, and store it in your google drive\n",
        "- ensure that you have combined.csv stored in your directory\n",
        "\n",
        "To train from scratch:\n",
        "1. [Import libraries](#scrollTo=6R9dFf2BZBUt&line=1&uniqifier=1)\n",
        "2. [Load the preprocessed dataset](#scrollTo=7_4lFKnZZH7e&line=1&uniqifier=1)\n",
        "3. [DistilRoberta Tokenization](#scrollTo=sW5r1HbZv2HL)\n",
        "4. [Define attention](#scrollTo=GzVX8jhbwQZw)\n",
        "5. [Define the decoder architecture](#scrollTo=pGzpY6f20quk)\n",
        "6. [Define loss function](#scrollTo=zAfJAgKFDKgN)\n",
        "7. [Define training functions](#scrollTo=f848ae33&line=9&uniqifier=1)\n",
        "8. [Define and finetune DistilRoberta encoder](#scrollTo=Urkzy-wexPqO)\n",
        "9. [Initialise and train model](#scrollTo=845aa1a3&line=5&uniqifier=1) (Set the loadFilename variable to None)\n",
        "10. [Run evaluation functions](#scrollTo=484e6fdc&line=6&uniqifier=1)\n",
        "11. [Evaluate with BERT, ROGUE, BLEU Score (quantitative)](#scrollTo=QiNsbbyrW-xR)\n",
        "12. [Host on Telegram](#scrollTo=icPY3cbo3ksU&line=1&uniqifier=1)\n",
        "\n",
        "To run saved model:\n",
        "1. [Import libraries](#scrollTo=6R9dFf2BZBUt&line=1&uniqifier=1)\n",
        "2. [Load the preprocessed dataset](#scrollTo=7_4lFKnZZH7e&line=1&uniqifier=1)\n",
        "3. [DistilRoberta Tokenization](#scrollTo=sW5r1HbZv2HL)\n",
        "4. [Define attention](#scrollTo=GzVX8jhbwQZw)\n",
        "5. [Define the decoder architecture](#scrollTo=pGzpY6f20quk)\n",
        "6. [Define loss function](#scrollTo=zAfJAgKFDKgN)\n",
        "7. [Define training functions](#scrollTo=f848ae33&line=9&uniqifier=1)\n",
        "8. [Define and finetune DistilRoberta encoder](#scrollTo=Urkzy-wexPqO)\n",
        "9. [Initialise and train model](#scrollTo=845aa1a3&line=5&uniqifier=1) (Set the loadFilename variable to your saved checkpoint file)\n",
        "10. [Run evaluation functions](#scrollTo=484e6fdc&line=6&uniqifier=1)\n",
        "11. [Evaluate with BERT, ROGUE, BLEU Score (quantitative)](#scrollTo=QiNsbbyrW-xR)\n",
        "12. [Host on Telegram](#scrollTo=icPY3cbo3ksU&line=1&uniqifier=1)"
      ],
      "metadata": {
        "id": "AMyrAGgx2jRz"
      },
      "id": "AMyrAGgx2jRz"
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Imports"
      ],
      "metadata": {
        "id": "6R9dFf2BZBUt"
      },
      "id": "6R9dFf2BZBUt"
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install random\n",
        "!pip install torch\n",
        "!pip install itertools\n",
        "!pip install pandas\n",
        "!pip install collections\n",
        "!pip install os\n",
        "!pip install transformer"
      ],
      "metadata": {
        "id": "WphBl1JcB7kk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a59a9bae-2cd6-47db-f470-8a9d7d9bd63a"
      },
      "id": "WphBl1JcB7kk",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[31mERROR: Could not find a version that satisfies the requirement random (from versions: none)\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[31mERROR: No matching distribution found for random\u001b[0m\u001b[31m\n",
            "\u001b[0mRequirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.5.1+cu121)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.16.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch) (4.12.2)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2024.10.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch) (1.3.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (3.0.2)\n",
            "\u001b[31mERROR: Could not find a version that satisfies the requirement itertools (from versions: none)\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[31mERROR: No matching distribution found for itertools\u001b[0m\u001b[31m\n",
            "\u001b[0mRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (2.2.2)\n",
            "Requirement already satisfied: numpy>=1.22.4 in /usr/local/lib/python3.10/dist-packages (from pandas) (1.26.4)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas) (2024.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n",
            "\u001b[31mERROR: Could not find a version that satisfies the requirement collections (from versions: none)\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[31mERROR: No matching distribution found for collections\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[31mERROR: Could not find a version that satisfies the requirement os (from versions: none)\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[31mERROR: No matching distribution found for os\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[31mERROR: Could not find a version that satisfies the requirement transformer (from versions: none)\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[31mERROR: No matching distribution found for transformer\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive', force_remount=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pKTNZmHWA0iW",
        "outputId": "16d075b0-486e-4bea-99fd-e452ad4c7327"
      },
      "id": "pKTNZmHWA0iW",
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Change FOLDER_PATH to the directory with combined_data.csv"
      ],
      "metadata": {
        "id": "cLeNf3KbvmP_"
      },
      "id": "cLeNf3KbvmP_"
    },
    {
      "cell_type": "code",
      "source": [
        "FOLDER_PATH = '/content/drive/My Drive/project'"
      ],
      "metadata": {
        "id": "zprgwoo8BLiw"
      },
      "id": "zprgwoo8BLiw",
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from torch.jit import script, trace\n",
        "import torch.nn as nn\n",
        "from torch import optim\n",
        "import torch.nn.functional as F\n",
        "import csv\n",
        "import random\n",
        "import re\n",
        "import os\n",
        "import unicodedata\n",
        "from io import open\n",
        "import itertools\n",
        "import math\n",
        "import json\n",
        "import pandas as pd\n",
        "\n",
        "\n",
        "\n",
        "USE_CUDA = torch.cuda.is_available()\n",
        "device = torch.device(\"cuda\" if USE_CUDA else \"cpu\")"
      ],
      "metadata": {
        "id": "F1kkMr5wBQz6"
      },
      "id": "F1kkMr5wBQz6",
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Loading Saved Dataset"
      ],
      "metadata": {
        "id": "7_4lFKnZZH7e"
      },
      "id": "7_4lFKnZZH7e"
    },
    {
      "cell_type": "code",
      "source": [
        "# read from csv\n",
        "df = pd.read_csv(f'{FOLDER_PATH}/combined_data.csv')\n",
        "print(df.head())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lpgeH3_yBUHy",
        "outputId": "ff39f64a-7ba4-4ef8-c3c2-eed1e5f0c56a"
      },
      "id": "lpgeH3_yBUHy",
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "                                 Question  \\\n",
            "0      how does randomised algorithm work   \n",
            "1    what do you mean by bestfirst search   \n",
            "2             how do you explain a daemon   \n",
            "3              what is phonetic algorithm   \n",
            "4  what do you mean by uniform costsearch   \n",
            "\n",
            "                                              Answer  \n",
            "0  the algorithm typically uses uniformly random ...  \n",
            "1  bestfirst search is a search algorithm which e...  \n",
            "2  daemon disk and execution monitor is a process...  \n",
            "3  a phonetic algorithm is an algorithm for index...  \n",
            "4  a tree search that finds the lowestcost route ...  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "0d3ce0f5",
      "metadata": {
        "id": "0d3ce0f5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        },
        "outputId": "ce3bb652-a083-468c-986a-815bdaf2f07f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                               Question  \\\n",
              "0                    how does randomised algorithm work   \n",
              "1                  what do you mean by bestfirst search   \n",
              "2                           how do you explain a daemon   \n",
              "3                            what is phonetic algorithm   \n",
              "4                what do you mean by uniform costsearch   \n",
              "...                                                 ...   \n",
              "3770                      explain biasvariance tradeoff   \n",
              "3771  what is stochastic gradient descent sgd in mac...   \n",
              "3772                explain stochastic gradient descent   \n",
              "3773  what is the backpropagation algorithm in machi...   \n",
              "3774                  explain backpropagation algorithm   \n",
              "\n",
              "                                                 Answer  \n",
              "0     the algorithm typically uses uniformly random ...  \n",
              "1     bestfirst search is a search algorithm which e...  \n",
              "2     daemon disk and execution monitor is a process...  \n",
              "3     a phonetic algorithm is an algorithm for index...  \n",
              "4     a tree search that finds the lowestcost route ...  \n",
              "...                                                 ...  \n",
              "3770  biasvariance tradeoff is a concept in machine ...  \n",
              "3771  stochastic gradient descent sgd is an optimiza...  \n",
              "3772  stochastic gradient descent sgd is an optimiza...  \n",
              "3773  the backpropagation algorithm is a widely used...  \n",
              "3774  the backpropagation algorithm is a widely used...  \n",
              "\n",
              "[3775 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6ab7265f-542e-4568-bc97-d208b29142af\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Question</th>\n",
              "      <th>Answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>how does randomised algorithm work</td>\n",
              "      <td>the algorithm typically uses uniformly random ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>what do you mean by bestfirst search</td>\n",
              "      <td>bestfirst search is a search algorithm which e...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>how do you explain a daemon</td>\n",
              "      <td>daemon disk and execution monitor is a process...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>what is phonetic algorithm</td>\n",
              "      <td>a phonetic algorithm is an algorithm for index...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>what do you mean by uniform costsearch</td>\n",
              "      <td>a tree search that finds the lowestcost route ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3770</th>\n",
              "      <td>explain biasvariance tradeoff</td>\n",
              "      <td>biasvariance tradeoff is a concept in machine ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3771</th>\n",
              "      <td>what is stochastic gradient descent sgd in mac...</td>\n",
              "      <td>stochastic gradient descent sgd is an optimiza...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3772</th>\n",
              "      <td>explain stochastic gradient descent</td>\n",
              "      <td>stochastic gradient descent sgd is an optimiza...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3773</th>\n",
              "      <td>what is the backpropagation algorithm in machi...</td>\n",
              "      <td>the backpropagation algorithm is a widely used...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3774</th>\n",
              "      <td>explain backpropagation algorithm</td>\n",
              "      <td>the backpropagation algorithm is a widely used...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3775 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6ab7265f-542e-4568-bc97-d208b29142af')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-6ab7265f-542e-4568-bc97-d208b29142af button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-6ab7265f-542e-4568-bc97-d208b29142af');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-a301c364-60e0-47f6-a51c-db0b5f941c86\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a301c364-60e0-47f6-a51c-db0b5f941c86')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-a301c364-60e0-47f6-a51c-db0b5f941c86 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_f16ba386-fa91-4f47-9750-de454fe69c7c\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_f16ba386-fa91-4f47-9750-de454fe69c7c button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 3775,\n  \"fields\": [\n    {\n      \"column\": \"Question\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3299,\n        \"samples\": [\n          \"define bidirectional search\",\n          \"what do you mean by divide and conquer algorithm\",\n          \"explain accumulatorbased architecture in computer systems\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Answer\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1817,\n        \"samples\": [\n          \"inner join returns matching rows from both tables left join returns all rows from the left table and matching rows from the right\",\n          \"the basic operations carried out in a linked list include creation of a listinsertion of a nodedeletion of a nodemodification of a nodetraversal of the list\",\n          \"there is no concept of folder in ubuntu everything included in your hardware is a file\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "df"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check max length for Question Answer pair"
      ],
      "metadata": {
        "id": "mlTgk7qKvxT-"
      },
      "id": "mlTgk7qKvxT-"
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "4c9be4b5",
      "metadata": {
        "id": "4c9be4b5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e9014cee-e337-4e81-a735-bf99e15854fc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The maximum length of an answer in terms of tokens is: 177\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Calculate the length of each answer\n",
        "df['answer_length'] = df['Answer'].apply(lambda x: len(x.split()))\n",
        "\n",
        "# Find the maximum length\n",
        "max_answer_length = df['answer_length'].max()\n",
        "\n",
        "print(f\"The maximum length of an answer in terms of tokens is: {max_answer_length}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "b1de00e5",
      "metadata": {
        "id": "b1de00e5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d098f49b-3f92-4e86-9ed6-e0efd4e0879c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The maximum length of an question in terms of tokens is: 105\n"
          ]
        }
      ],
      "source": [
        "df['question_length'] = df['Question'].apply(lambda x: len(x.split()))\n",
        "\n",
        "max_question_length = df['question_length'].max()\n",
        "\n",
        "print(f\"The maximum length of an question in terms of tokens is: {max_question_length}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Tokenization using DistilRoberta Tokenizer**"
      ],
      "metadata": {
        "id": "sW5r1HbZv2HL"
      },
      "id": "sW5r1HbZv2HL"
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "acbdaab7",
      "metadata": {
        "id": "acbdaab7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 554,
          "referenced_widgets": [
            "be531bb8ef274e34925835b254756cc2",
            "02ca0cf92e5c420d8b9d9214d18eec0c",
            "201922e1c2e2419dbe1473c192d7f864",
            "831c6f9492d9491fa07c8d14263b23c6",
            "2d53b8369c7c4d25995b45e7df2f39ac",
            "cb5d5a61ac484e62ac862ae2e8c4d35b",
            "5bc766a4b59640c4ad4175f44b3c6114",
            "91e9160c803740d183082fbbbd285437",
            "348c563c16a648d68569a967c7dd0d20",
            "56d8b599a7dd4c0dbf00483ab7d06f39",
            "c07be4243736443bad0090f05f7df2e1",
            "7ac28328549d428aab0c715116d3579f",
            "a1c3b002f9a543dfbc05dfa880fc934f",
            "b2ff67d69ba142bb95e7d4c8388a6935",
            "2be8dde9528445fbae429a25988f228d",
            "c08fe52e280d4cab903ab36f4a550238",
            "8ea51d2da3a345a49990d5cb9839c073",
            "e390578f1e6f4e04aabadb35b03ea5d8",
            "2668a12bb7ad49b8824c6c3240e82ce5",
            "1c7e94e5c08943788c1fdf1a5c40749b",
            "ea5a79e74de741cca7f0af1fe7cc2f1f",
            "ccfac9b66fea4c4a835a03f809aeca6a",
            "0c6ee3ce8a68494c82298201606e3786",
            "adb966239e61491791998da41c830d30",
            "99b3437b804e45808a21ab5c8c6fd84d",
            "c092685faef545f2837a2230495e4e1c",
            "9bfeb9640bde4c57b082e1007fec6d4e",
            "f1780a56fc5640279e9a31b4f909d152",
            "5f14cea471404f5da11243f097020ede",
            "1e54eb3786ec44729ed29a06ff8e402a",
            "90a6ec622d8d4106abc3e1f7b224675d",
            "568e2f77748241b6811ef2ae6196f116",
            "d71571cc227248729352a1ac258adc30",
            "73e9503f09204413a5d4a511e7d8822f",
            "ae259abd8a7b404296b9c6df42513d22",
            "663a6ca2b1404a48a1510e42429a4d17",
            "89726182aeac446eb7a52d6d423432ec",
            "39305bc8add642b5866fe4dd93581762",
            "c27490a375864c8b89fce811796249c0",
            "a5273db64fa94e5b8c643094557bc9a2",
            "a754a76d0a7e42d1ad7940f3c410ae25",
            "9d452f9d7f4845c2bee5b0fd729b89f3",
            "4631ae7f63fa41d3bb499dfdc532489a",
            "21b6b90a90254ecd895cac314b611a4f",
            "c5178b2628ae4378b8659d7aae5e8b3f",
            "51c00a71b3554f8985b43abf988192a4",
            "4a69ab073e03496f898310e1fdaeb43e",
            "1662089c61324150b997413129eeae7f",
            "bbfa7628c1434b08887a32bd445dacf0",
            "9b1d7d1c2ce743c09b5a2324c11ba7e1",
            "dfae4c8fa454402faec1a824afeb4bc0",
            "99c6857bce134261949392f4ca0c3ccb",
            "036bf95bfcef4707bbb11639f1d633bb",
            "353a55607f7f4a628d94024e01c000cc",
            "2bd9dc5b40f44d10a2e8097a83147a3c"
          ]
        },
        "outputId": "5ef5c5d9-4d4f-4210-dc1b-7ad3f8aebd14"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/25.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "be531bb8ef274e34925835b254756cc2"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/480 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7ac28328549d428aab0c715116d3579f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "vocab.json:   0%|          | 0.00/899k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0c6ee3ce8a68494c82298201606e3786"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "73e9503f09204413a5d4a511e7d8822f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c5178b2628ae4378b8659d7aae5e8b3f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target vocabulary size (output_size): 6433\n",
            "tensor([[    0,  9178,   473,  ...,     1,     1,     1],\n",
            "        [    0, 12196,   109,  ...,     1,     1,     1],\n",
            "        [    0,  9178,   109,  ...,     1,     1,     1],\n",
            "        ...,\n",
            "        [    0, 23242,  1851,  ...,     1,     1,     1],\n",
            "        [    0, 12196,    16,  ...,     1,     1,     1],\n",
            "        [    0, 23242,  1851,  ...,     1,     1,     1]])\n",
            "tensor([[    0,   627, 17194,  ...,     1,     1,     1],\n",
            "        [    0,  7885,  9502,  ...,     1,     1,     1],\n",
            "        [    0,  6106, 34344,  ...,     1,     1,     1],\n",
            "        ...,\n",
            "        [    0,   620,  4306,  ...,     1,     1,     1],\n",
            "        [    0,   627,   124,  ...,     1,     1,     1],\n",
            "        [    0,   627,   124,  ...,     1,     1,     1]])\n"
          ]
        }
      ],
      "source": [
        "from transformers import AutoTokenizer\n",
        "import pandas as pd\n",
        "from collections import Counter\n",
        "\n",
        "# Load the DistilRoBERTa tokenizer\n",
        "tokenizer = AutoTokenizer.from_pretrained('distilroberta-base')\n",
        "\n",
        "questions = df['Question'].tolist()\n",
        "answers = df['Answer'].tolist()\n",
        "\n",
        "MAX_LENGTH = 180\n",
        "\n",
        "# Tokenize, pad, and truncate the questions\n",
        "question_encodings = tokenizer(\n",
        "    questions,\n",
        "    padding=True,  # pad to the longest sequence in the batch\n",
        "    truncation=True,  # Truncate sequences to the max_length\n",
        "    max_length=MAX_LENGTH,\n",
        "    return_tensors='pt'\n",
        ")\n",
        "\n",
        "answer_encodings = tokenizer(\n",
        "    answers,\n",
        "    padding=True,\n",
        "    truncation=True,\n",
        "    max_length=MAX_LENGTH,\n",
        "    return_tensors='pt'\n",
        ")\n",
        "\n",
        "all_tokens = []\n",
        "for answer in answers:\n",
        "    tokens = tokenizer.tokenize(answer)\n",
        "    all_tokens.extend(tokens)\n",
        "\n",
        "for qn in questions:\n",
        "    tokens = tokenizer.tokenize(qn)\n",
        "    all_tokens.extend(tokens)\n",
        "\n",
        "vocab_counter = Counter(all_tokens)\n",
        "total_tokens = len(vocab_counter)\n",
        "\n",
        "print(f\"Target vocabulary size (output_size): {total_tokens}\")\n",
        "\n",
        "# Display the tokenized and padded data\n",
        "print(question_encodings['input_ids'])\n",
        "print(answer_encodings['input_ids'])"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "X = question_encodings['input_ids']\n",
        "attention_mask_X = question_encodings['attention_mask']\n",
        "y = answer_encodings['input_ids']\n",
        "attention_mask_y = answer_encodings['attention_mask']\n",
        "\n",
        "X_train = X\n",
        "attention_mask_X_train = attention_mask_X\n",
        "y_train = y\n",
        "attention_mask_y_train = attention_mask_y"
      ],
      "metadata": {
        "id": "M0oYciEL_GS3"
      },
      "id": "M0oYciEL_GS3",
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data import DataLoader, Dataset\n",
        "\n",
        "class Seq2SeqDataset(Dataset):\n",
        "    def __init__(self, input_data, target_data):\n",
        "        self.input_data = input_data\n",
        "        self.target_data = target_data\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.input_data['input_ids'])\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        return {\n",
        "            'input_ids': self.input_data['input_ids'][idx],\n",
        "            'attention_mask': self.input_data['attention_mask'][idx],\n",
        "            'target_ids': self.target_data['input_ids'][idx],\n",
        "            'target_attention_mask': self.target_data['attention_mask'][idx]\n",
        "        }\n",
        "\n",
        "train_dataset = Seq2SeqDataset(\n",
        "    {'input_ids': X_train, 'attention_mask': attention_mask_X_train},\n",
        "    {'input_ids': y_train, 'attention_mask': attention_mask_y_train}\n",
        ")\n",
        "\n",
        "# Create a DataLoader for the full dataset\n",
        "data_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)\n",
        "print(f\"size: {len(data_loader)}\")"
      ],
      "metadata": {
        "id": "k6EJhhYS_Riu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b016dd98-de2a-4261-d0df-bcb21cfbfece"
      },
      "id": "k6EJhhYS_Riu",
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "size: 59\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Luong Attention**"
      ],
      "metadata": {
        "id": "GzVX8jhbwQZw"
      },
      "id": "GzVX8jhbwQZw"
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "13097c2c",
      "metadata": {
        "id": "13097c2c"
      },
      "outputs": [],
      "source": [
        "# Luong attention layer\n",
        "class Attn(torch.nn.Module):\n",
        "    def __init__(self, method, hidden_size):\n",
        "        super(Attn, self).__init__()\n",
        "        self.method = method\n",
        "        self.hidden_size = hidden_size\n",
        "        if self.method == 'general':\n",
        "            self.attn = torch.nn.Linear(self.hidden_size, hidden_size)\n",
        "        elif self.method == 'concat':\n",
        "            self.attn = torch.nn.Linear(self.hidden_size * 2, hidden_size)\n",
        "            self.v = torch.nn.Parameter(torch.FloatTensor(hidden_size))\n",
        "\n",
        "    def dot_score(self, hidden, encoder_output):\n",
        "        return torch.sum(hidden * encoder_output, dim=2)\n",
        "\n",
        "    def general_score(self, hidden, encoder_output):\n",
        "        energy = self.attn(encoder_output)\n",
        "        return torch.sum(hidden * energy, dim=2)\n",
        "\n",
        "    def concat_score(self, hidden, encoder_output):\n",
        "        energy = self.attn(torch.cat((hidden.expand(encoder_output.size(0), -1, -1), encoder_output), 2)).tanh()\n",
        "        return torch.sum(self.v * energy, dim=2)\n",
        "\n",
        "    def forward(self, hidden, encoder_outputs):\n",
        "        if self.method == 'general':\n",
        "            attn_energies = self.general_score(hidden, encoder_outputs)\n",
        "        elif self.method == 'concat':\n",
        "            attn_energies = self.concat_score(hidden, encoder_outputs)\n",
        "        elif self.method == 'dot':\n",
        "            attn_energies = self.dot_score(hidden, encoder_outputs)\n",
        "\n",
        "        # Transpose max_length and batch_size dimensions\n",
        "        attn_energies = attn_energies.t()\n",
        "\n",
        "        # Return the softmax normalized probability scores (with added dimension)\n",
        "        return F.softmax(attn_energies, dim=1).unsqueeze(1) # (batch_size,1,max_length)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**LSTM Decoder**"
      ],
      "metadata": {
        "id": "pGzpY6f20quk"
      },
      "id": "pGzpY6f20quk"
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "c7646da1",
      "metadata": {
        "id": "c7646da1"
      },
      "outputs": [],
      "source": [
        "import torch.nn as nn\n",
        "\n",
        "\n",
        "class LuongAttnDecoderLSTM(nn.Module):\n",
        "    def __init__(self, attn_model, embedding, hidden_size, output_size, n_layers=1, dropout=0.1):\n",
        "        super(LuongAttnDecoderLSTM, self).__init__()\n",
        "        self.attn_model = attn_model\n",
        "        self.hidden_size = hidden_size\n",
        "        self.output_size = output_size\n",
        "        self.n_layers = n_layers\n",
        "        self.dropout = dropout\n",
        "\n",
        "        self.embedding = embedding\n",
        "        self.embedding_dropout = nn.Dropout(dropout)\n",
        "        self.lstm = nn.LSTM(hidden_size, hidden_size, n_layers, dropout=(0 if n_layers == 1 else dropout))\n",
        "        self.concat = nn.Linear(hidden_size * 2, hidden_size)\n",
        "        self.out = nn.Linear(hidden_size, output_size)\n",
        "\n",
        "        self.attn = Attn(attn_model, hidden_size)\n",
        "\n",
        "    def forward(self, input_seq, last_hidden, encoder_outputs):\n",
        "        embedded = self.embedding(input_seq)\n",
        "        embedded = self.embedding_dropout(embedded)\n",
        "        rnn_output, hidden = self.lstm(embedded, last_hidden)\n",
        "\n",
        "        # Calculate attention weights from the current LSTM output\n",
        "        attn_weights = self.attn(rnn_output, encoder_outputs)\n",
        "\n",
        "        context = attn_weights.bmm(encoder_outputs.transpose(0, 1))\n",
        "        rnn_output = rnn_output.squeeze(0)\n",
        "        context = context.squeeze(1)\n",
        "        concat_input = torch.cat((rnn_output, context), 1)\n",
        "        concat_output = torch.tanh(self.concat(concat_input))\n",
        "        # Predict next word using Luong eq. 6\n",
        "        output = self.out(concat_output)\n",
        "        output = F.softmax(output, dim=1)\n",
        "        return output, hidden"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b>Masked NLL Loss</b>\n",
        "- employed to focus loss calculation on relevant parts of output sequence, ignoring padding tokens"
      ],
      "metadata": {
        "id": "zAfJAgKFDKgN"
      },
      "id": "zAfJAgKFDKgN"
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "6a173af9",
      "metadata": {
        "id": "6a173af9"
      },
      "outputs": [],
      "source": [
        "def maskNLLLoss(inp, target, mask):\n",
        "\n",
        "    nTotal = mask.sum()  # Total number of valid elements\n",
        "\n",
        "    if nTotal == 0:\n",
        "        return torch.tensor(0.0, device=device), 0  # Handle case with no valid elements\n",
        "\n",
        "    # Add epsilon to avoid log(0)\n",
        "    epsilon = 1e-10\n",
        "    crossEntropy = -torch.log(torch.gather(inp, 1, target.view(-1, 1)) + epsilon)\n",
        "\n",
        "    # Apply mask and compute mean loss\n",
        "    loss = crossEntropy.masked_select(mask).mean()\n",
        "\n",
        "    return loss.to(device), nTotal.item()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Teacher Forcing**\n",
        "- employed to help the model learn more effectively by providing it with the correct context"
      ],
      "metadata": {
        "id": "7TUR3m1q05mD"
      },
      "id": "7TUR3m1q05mD"
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "f848ae33",
      "metadata": {
        "id": "f848ae33"
      },
      "outputs": [],
      "source": [
        "def train(input_variable, attention_mask, target_variable, target_attention_mask, encoder, decoder, embedding,\n",
        "          encoder_optimizer, decoder_optimizer, batch_size, clip, max_target_len=MAX_LENGTH):\n",
        "\n",
        "    # Zero gradients\n",
        "    encoder_optimizer.zero_grad()\n",
        "    decoder_optimizer.zero_grad()\n",
        "\n",
        "    loss = 0\n",
        "    print_losses = []\n",
        "    n_totals = 0\n",
        "\n",
        "    # Forward pass through encoder\n",
        "    encoder_outputs, encoder_last_hidden = encoder(input_ids=input_variable, attention_mask=attention_mask)\n",
        "    decoder_hidden = encoder_last_hidden\n",
        "\n",
        "    SOS_token = \"<s>\"\n",
        "    SOS_token_id = tokenizer.convert_tokens_to_ids(SOS_token)\n",
        "\n",
        "    decoder_input = torch.LongTensor([[SOS_token_id] * batch_size]).to(device)\n",
        "\n",
        "    # Determine if we are using teacher forcing this iteration\n",
        "    use_teacher_forcing = True if random.random() < teacher_forcing_ratio else False\n",
        "\n",
        "    # Forward batch of sequences through decoder one time step at a time\n",
        "    if use_teacher_forcing:\n",
        "        for t in range(max_target_len):\n",
        "            decoder_output, decoder_hidden = decoder(\n",
        "                decoder_input, decoder_hidden, encoder_outputs\n",
        "            )\n",
        "\n",
        "            # Teacher forcing: next input is current target\n",
        "            decoder_input = target_variable[t].view(1, -1)\n",
        "            # Calculate and accumulate loss\n",
        "            mask_loss, nTotal = maskNLLLoss(decoder_output, target_variable[t], target_attention_mask[t])\n",
        "            loss += mask_loss\n",
        "            print_losses.append(mask_loss.item() * nTotal)\n",
        "            n_totals += nTotal\n",
        "    else:\n",
        "        for t in range(max_target_len):\n",
        "            decoder_output, decoder_hidden = decoder(\n",
        "                decoder_input, decoder_hidden, encoder_outputs\n",
        "            )\n",
        "            # No teacher forcing: next input is decoder's own current output\n",
        "            _, topi = decoder_output.topk(1)\n",
        "            decoder_input = torch.LongTensor([[topi[i][0] for i in range(batch_size)]])\n",
        "            decoder_input = decoder_input.to(device)\n",
        "            # Calculate and accumulate loss\n",
        "            mask_loss, nTotal = maskNLLLoss(decoder_output, target_variable[t], target_attention_mask[t])\n",
        "            loss += mask_loss\n",
        "            print_losses.append(mask_loss.item() * nTotal)\n",
        "            n_totals += nTotal\n",
        "\n",
        "    # Perform backpropatation\n",
        "    loss.backward()\n",
        "\n",
        "    # Clip gradients: gradients are modified in place\n",
        "    _ = torch.nn.utils.clip_grad_norm_(encoder.parameters(), clip)\n",
        "    _ = torch.nn.utils.clip_grad_norm_(decoder.parameters(), clip)\n",
        "\n",
        "    # Adjust model weights\n",
        "    encoder_optimizer.step()\n",
        "    decoder_optimizer.step()\n",
        "\n",
        "    return sum(print_losses) / n_totals"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def trainIters(model_name, encoder, decoder, encoder_optimizer, decoder_optimizer, embedding, decoder_n_layers, save_dir, n_iteration, batch_size, print_every, save_every, clip, loadFilename):\n",
        "\n",
        "    print('Initializing ...')\n",
        "    start_iteration = 1\n",
        "    print_loss = 0\n",
        "    if loadFilename:\n",
        "        start_iteration = checkpoint['iteration'] + 1\n",
        "\n",
        "    # Training loop\n",
        "    print(\"Training...\")\n",
        "    vocab_size = tokenizer.vocab_size\n",
        "    if hasattr(decoder, 'output_size'):\n",
        "        assert decoder.output_size == vocab_size, \"Model output size does not match vocabulary size!\"\n",
        "    else:\n",
        "        decoder = nn.Linear(hidden_size, vocab_size)\n",
        "\n",
        "    iteration = start_iteration\n",
        "\n",
        "    while iteration <= n_iteration:\n",
        "        for batch in data_loader:\n",
        "            if iteration > n_iteration:\n",
        "                break\n",
        "            input_variable = batch['input_ids'].to(device)\n",
        "            attention_mask = batch['attention_mask'].to(device)\n",
        "            target_variable = batch['target_ids'].to(device)\n",
        "            target_attention_mask = batch['target_attention_mask'].to(device)\n",
        "            target_variable = target_variable.transpose(0, 1)\n",
        "            target_attention_mask = target_attention_mask.transpose(0, 1)\n",
        "            target_attention_mask = target_attention_mask.bool()\n",
        "            batch_size = input_variable.size(0)\n",
        "\n",
        "            # Run a training iteration with batch\n",
        "            loss = train(input_variable, attention_mask, target_variable, target_attention_mask, encoder, decoder, embedding, encoder_optimizer, decoder_optimizer, batch_size, clip)\n",
        "            print_loss += loss\n",
        "\n",
        "            # Print progress\n",
        "            if iteration % print_every == 0:\n",
        "                print_loss_avg = print_loss / print_every\n",
        "                print(\"Iteration: {}; Percent complete: {:.1f}%; Average loss: {:.4f}\".format(iteration, iteration / n_iteration * 100, print_loss_avg))\n",
        "                print_loss = 0\n",
        "\n",
        "            # Save checkpoint\n",
        "            if iteration % save_every == 0:\n",
        "                directory = os.path.join(save_dir, model_name, '{}_{}'.format(decoder_n_layers, hidden_size))\n",
        "                if not os.path.exists(directory):\n",
        "                    os.makedirs(directory)\n",
        "                torch.save({\n",
        "                    'iteration': iteration,\n",
        "                    'en': encoder.state_dict(),\n",
        "                    'de': decoder.state_dict(),\n",
        "                    'en_opt': encoder_optimizer.state_dict(),\n",
        "                    'de_opt': decoder_optimizer.state_dict(),\n",
        "                    'loss': loss,\n",
        "                    'embedding': embedding.state_dict()\n",
        "                }, os.path.join(directory, '{}_{}.tar'.format(iteration, 'checkpoint')))\n",
        "\n",
        "            iteration += 1  # Increment iteration counter"
      ],
      "metadata": {
        "id": "nE0GaOKb_yB9"
      },
      "id": "nE0GaOKb_yB9",
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b>DistilRoberta Encoder</b>\n",
        "- DistilRoberta is finetuned before joint training to adapt to our CS Interview QA task"
      ],
      "metadata": {
        "id": "Urkzy-wexPqO"
      },
      "id": "Urkzy-wexPqO"
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "from transformers import (\n",
        "    AutoModelForMaskedLM,\n",
        "    AutoTokenizer,\n",
        "    DataCollatorForLanguageModeling,\n",
        "    Trainer,\n",
        "    TrainingArguments\n",
        ")\n",
        "\n",
        "class EncoderDistilRoberta(nn.Module):\n",
        "    def __init__(self, hidden_size, n_layers=2, dropout=0.1, freeze_layers=None):\n",
        "        super(EncoderDistilRoberta, self).__init__()\n",
        "        self.n_layers = n_layers\n",
        "        self.hidden_size = hidden_size\n",
        "\n",
        "        # Load the DistilRoBERTa model\n",
        "        self.distilroberta = AutoModelForMaskedLM.from_pretrained('distilroberta-base')\n",
        "\n",
        "        # Freeze specified layers if needed\n",
        "        if freeze_layers is not None:\n",
        "            for name, param in self.distilroberta.named_parameters():\n",
        "                if any(str(layer) in name for layer in freeze_layers):\n",
        "                    param.requires_grad = False\n",
        "\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.hidden_transform = nn.Linear(768, hidden_size)\n",
        "        self.layer_norm = nn.LayerNorm(hidden_size)\n",
        "\n",
        "    def forward(self, input_ids, attention_mask):\n",
        "        outputs = self.distilroberta.roberta(input_ids=input_ids,\n",
        "                                           attention_mask=attention_mask)\n",
        "\n",
        "        hidden_states = outputs.last_hidden_state\n",
        "        hidden_states = self.dropout(hidden_states)\n",
        "        # Transform hidden states to make it suitable for input to LSTM decoder\n",
        "        encoder_output = hidden_states.permute(1, 0, 2)\n",
        "        encoder_output = self.hidden_transform(encoder_output)\n",
        "        encoder_output = self.layer_norm(encoder_output)\n",
        "\n",
        "        last_hidden_state = hidden_states[:, -1, :]\n",
        "        transformed_last_hidden_state = self.hidden_transform(last_hidden_state)\n",
        "        decoder_hidden = transformed_last_hidden_state.unsqueeze(0).repeat(self.n_layers, 1, 1)\n",
        "        # Initialize cell state as zeros as DistilRoberta does not have an internal cell state\n",
        "        cell_state = torch.zeros_like(decoder_hidden)\n",
        "\n",
        "        return encoder_output, (decoder_hidden, cell_state)\n"
      ],
      "metadata": {
        "id": "FvaejtWSFEhk"
      },
      "id": "FvaejtWSFEhk",
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Convert dataset to MLM format for DistilRoberta Finetuning\n",
        "- MLM Dataset helps models learn contextual relationships between words, allowing it to develop a deeper understanding of language structure, semantics and syntax"
      ],
      "metadata": {
        "id": "sT4rpjmnz6th"
      },
      "id": "sT4rpjmnz6th"
    },
    {
      "cell_type": "code",
      "source": [
        "# convert Seq2SeqDataset to MLM format\n",
        "def create_mlm_dataset(dataset):\n",
        "    max_length = MAX_LENGTH\n",
        "    mlm_inputs = []\n",
        "    mlm_masks = []\n",
        "    tokenizer = AutoTokenizer.from_pretrained('distilroberta-base')\n",
        "    pad_token_id = tokenizer.pad_token_id\n",
        "\n",
        "    for i in range(len(dataset)):\n",
        "        item = dataset[i]\n",
        "        q_input = item['input_ids']\n",
        "        q_mask = item['attention_mask']\n",
        "        q_pad_length = max_length - q_input.size(0)\n",
        "        if q_pad_length > 0:\n",
        "            q_input = torch.cat([q_input, torch.full((q_pad_length,), pad_token_id)])\n",
        "            q_mask = torch.cat([q_mask, torch.zeros(q_pad_length)])\n",
        "\n",
        "        a_input = item['target_ids']\n",
        "        a_mask = item['target_attention_mask']\n",
        "        a_pad_length = max_length - a_input.size(0)\n",
        "        if a_pad_length > 0:\n",
        "            a_input = torch.cat([a_input, torch.full((a_pad_length,), pad_token_id)])\n",
        "            a_mask = torch.cat([a_mask, torch.zeros(a_pad_length)])\n",
        "\n",
        "        mlm_inputs.extend([q_input, a_input])\n",
        "        mlm_masks.extend([q_mask, a_mask])\n",
        "\n",
        "    return {\n",
        "        'input_ids': torch.stack(mlm_inputs),\n",
        "        'attention_mask': torch.stack(mlm_masks),\n",
        "        'labels': torch.stack(mlm_inputs).clone()\n",
        "    }"
      ],
      "metadata": {
        "id": "RetZiYv4yesX"
      },
      "id": "RetZiYv4yesX",
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "DistilRoberta Fine-Tuning"
      ],
      "metadata": {
        "id": "rhOSSbxB0OoC"
      },
      "id": "rhOSSbxB0OoC"
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the directory path in Google Drive\n",
        "output_dir = f'{FOLDER_PATH}/distilroberta_fine_tuned_results'\n",
        "\n",
        "# Check if the directory exists\n",
        "if not os.path.exists(output_dir):\n",
        "    os.makedirs(output_dir)\n",
        "    print(f\"Directory '{output_dir}' created.\")\n",
        "else:\n",
        "    print(f\"Directory '{output_dir}' already exists.\")\n",
        "\n",
        "def fine_tune_encoder_with_existing_data(\n",
        "    dataset,\n",
        "    hidden_size=768,\n",
        "    output_dir=output_dir + '/results',\n",
        "    freeze_layers=None,\n",
        "    num_epochs=3,\n",
        "    batch_size=16,\n",
        "    learning_rate=2e-5\n",
        "):\n",
        "    # Initialize DistilRoberta model\n",
        "    encoder = EncoderDistilRoberta(\n",
        "        hidden_size=hidden_size,\n",
        "        freeze_layers=freeze_layers\n",
        "    )\n",
        "\n",
        "    # Convert dataset to MLM format\n",
        "    print(\"Creating MLM dataset...\")\n",
        "    mlm_data = create_mlm_dataset(dataset)\n",
        "    print(\"MLM dataset created successfully\")\n",
        "\n",
        "    # Create a custom Dataset class for the MLM data\n",
        "    class MLMDataset(Dataset):\n",
        "        def __init__(self, data):\n",
        "            self.data = data\n",
        "\n",
        "        def __len__(self):\n",
        "            return len(self.data['input_ids'])\n",
        "\n",
        "        def __getitem__(self, idx):\n",
        "            return {\n",
        "                'input_ids': self.data['input_ids'][idx],\n",
        "                'attention_mask': self.data['attention_mask'][idx],\n",
        "                'labels': self.data['labels'][idx]\n",
        "            }\n",
        "\n",
        "    # Convert to proper dataset\n",
        "    mlm_dataset = MLMDataset(mlm_data)\n",
        "\n",
        "    # Create training arguments\n",
        "    training_args = TrainingArguments(\n",
        "        output_dir=output_dir,\n",
        "        num_train_epochs=num_epochs,\n",
        "        per_device_train_batch_size=batch_size,\n",
        "        per_device_eval_batch_size=batch_size,\n",
        "        learning_rate=learning_rate,\n",
        "        warmup_ratio=0.1,\n",
        "        weight_decay=0.01,\n",
        "        gradient_accumulation_steps=4,\n",
        "        logging_dir=output_dir + '/logs',\n",
        "        logging_steps=100,\n",
        "        save_strategy='steps',\n",
        "        save_steps=500,\n",
        "        report_to=[],\n",
        "        fp16=True\n",
        "    )\n",
        "\n",
        "    # Create data collator for dynamic masking\n",
        "    tokenizer = AutoTokenizer.from_pretrained('distilroberta-base')\n",
        "    data_collator = DataCollatorForLanguageModeling(\n",
        "        tokenizer=tokenizer,\n",
        "        mlm=True,\n",
        "        mlm_probability=0.15\n",
        "    )\n",
        "\n",
        "    # Initialize trainer\n",
        "    trainer = Trainer(\n",
        "        model=encoder.distilroberta,\n",
        "        args=training_args,\n",
        "        train_dataset=mlm_dataset,\n",
        "        data_collator=data_collator\n",
        "    )\n",
        "\n",
        "    # Train the model\n",
        "    trainer.train()\n",
        "\n",
        "    return encoder\n",
        "\n",
        "def fine_tune_and_prepare_encoder(train_dataset, hidden_size=768):\n",
        "    # Fine-tune the encoder\n",
        "    encoder = fine_tune_encoder_with_existing_data(\n",
        "        dataset=train_dataset,\n",
        "        # val_dataset=val_dataset,\n",
        "        hidden_size=hidden_size,\n",
        "        output_dir=output_dir + '/results',\n",
        "        freeze_layers=[0, 1]  # Optionally freeze first two layers\n",
        "    )\n",
        "\n",
        "    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "    encoder = encoder.to(device)\n",
        "\n",
        "    return encoder"
      ],
      "metadata": {
        "id": "8L3w1RbP0LwT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7f29177c-ded2-48f5-b65a-991445ad5335"
      },
      "id": "8L3w1RbP0LwT",
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Directory '/content/drive/MyDrive/project/distilroberta_fine_tuned_results' created.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "encoder = fine_tune_and_prepare_encoder(\n",
        "    train_dataset=train_dataset,\n",
        "    hidden_size=768\n",
        ")"
      ],
      "metadata": {
        "id": "Hq2psO8BAdJW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 273
        },
        "outputId": "247fde79-6431-4c02-e405-7dc38b894e97"
      },
      "id": "Hq2psO8BAdJW",
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at distilroberta-base were not used when initializing RobertaForMaskedLM: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']\n",
            "- This IS expected if you are initializing RobertaForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Creating MLM dataset...\n",
            "MLM dataset created successfully\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='354' max='354' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [354/354 03:44, Epoch 3/3]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>100</td>\n",
              "      <td>3.164400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>200</td>\n",
              "      <td>2.731500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>300</td>\n",
              "      <td>2.696500</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "id": "845aa1a3",
      "metadata": {
        "id": "845aa1a3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9385dbe2-9ea2-43b6-bb48-39e90f65a558"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Building encoder and decoder ...\n",
            "Models built and ready to go!\n"
          ]
        }
      ],
      "source": [
        "from transformers import AutoModel\n",
        "\n",
        "# Configure models\n",
        "model_name = 'cb_latestBESTESTfullset_model'\n",
        "attn_model = 'dot'\n",
        "hidden_size = 768\n",
        "decoder_n_layers = 2\n",
        "dropout = 0.1\n",
        "batch_size = 64\n",
        "\n",
        "# Set checkpoint to load from; set to None if starting from scratch\n",
        "loadFilename = None\n",
        "checkpoint_iter = 4000\n",
        "# loadFilename = os.path.join(save_dir, model_name, corpus_name,\n",
        "#                             '{}-{}_{}'.format(encoder_n_layers, decoder_n_layers, hidden_size),\n",
        "#                             '{}_checkpoint.tar'.format(checkpoint_iter))\n",
        "\n",
        "\n",
        "# Load model if a loadFilename is provided\n",
        "if loadFilename:\n",
        "    # If loading on same machine the model was trained on\n",
        "    checkpoint = torch.load(loadFilename)\n",
        "    # If loading a model trained on GPU to CPU\n",
        "    #checkpoint = torch.load(loadFilename, map_location=torch.device('cpu'))\n",
        "    encoder_sd = checkpoint['en']\n",
        "    decoder_sd = checkpoint['de']\n",
        "    encoder_optimizer_sd = checkpoint['en_opt']\n",
        "    decoder_optimizer_sd = checkpoint['de_opt']\n",
        "    embedding_sd = checkpoint['embedding']\n",
        "#     voc.__dict__ = checkpoint['voc_dict']\n",
        "\n",
        "\n",
        "print('Building encoder and decoder ...')\n",
        "# Initialize word embeddings\n",
        "embedding = nn.Embedding(tokenizer.vocab_size, hidden_size)\n",
        "if loadFilename:\n",
        "    embedding.load_state_dict(embedding_sd)\n",
        "# Initialize encoder & decoder models\n",
        "decoder = LuongAttnDecoderLSTM(attn_model, embedding, hidden_size, tokenizer.vocab_size, decoder_n_layers, dropout)\n",
        "if loadFilename:\n",
        "    encoder.load_state_dict(encoder_sd)\n",
        "    decoder.load_state_dict(decoder_sd)\n",
        "# Use appropriate device\n",
        "encoder = encoder.to(device)\n",
        "decoder = decoder.to(device)\n",
        "print('Models built and ready to go!')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Start training the DistilRoberta-LSTM Model**"
      ],
      "metadata": {
        "id": "BeVoe3II1SXg"
      },
      "id": "BeVoe3II1SXg"
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "id": "eb40c516",
      "metadata": {
        "scrolled": true,
        "id": "eb40c516",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ceb06687-b2d4-41cc-beaa-b035119984ba"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Building optimizers ...\n",
            "Starting Training!\n",
            "Directory '/content/drive/MyDrive/project/distilroberta_checkpoints' already exists.\n",
            "Initializing ...\n",
            "Training...\n",
            "Iteration: 1; Percent complete: 5.0%; Average loss: 4.3275\n",
            "Iteration: 2; Percent complete: 10.0%; Average loss: 4.0685\n",
            "Iteration: 3; Percent complete: 15.0%; Average loss: 4.0079\n",
            "Iteration: 4; Percent complete: 20.0%; Average loss: 4.0082\n",
            "Iteration: 5; Percent complete: 25.0%; Average loss: 4.0911\n",
            "Iteration: 6; Percent complete: 30.0%; Average loss: 4.0113\n",
            "Iteration: 7; Percent complete: 35.0%; Average loss: 4.1782\n",
            "Iteration: 8; Percent complete: 40.0%; Average loss: 3.8778\n",
            "Iteration: 9; Percent complete: 45.0%; Average loss: 4.3307\n",
            "Iteration: 10; Percent complete: 50.0%; Average loss: 4.0300\n",
            "Iteration: 11; Percent complete: 55.0%; Average loss: 4.0757\n",
            "Iteration: 12; Percent complete: 60.0%; Average loss: 3.7002\n",
            "Iteration: 13; Percent complete: 65.0%; Average loss: 3.5559\n",
            "Iteration: 14; Percent complete: 70.0%; Average loss: 4.1236\n",
            "Iteration: 15; Percent complete: 75.0%; Average loss: 3.9987\n",
            "Iteration: 16; Percent complete: 80.0%; Average loss: 3.9452\n",
            "Iteration: 17; Percent complete: 85.0%; Average loss: 3.9374\n",
            "Iteration: 18; Percent complete: 90.0%; Average loss: 3.8068\n",
            "Iteration: 19; Percent complete: 95.0%; Average loss: 3.9561\n",
            "Iteration: 20; Percent complete: 100.0%; Average loss: 3.5640\n"
          ]
        }
      ],
      "source": [
        "from torch import optim\n",
        "import os\n",
        "\n",
        "\n",
        "# Configure training/optimization\n",
        "clip = 50.0\n",
        "teacher_forcing_ratio = 1.0\n",
        "learning_rate = 0.0001\n",
        "decoder_learning_ratio = 5.0\n",
        "n_iteration = 1000\n",
        "print_every = 1\n",
        "save_every = 500\n",
        "\n",
        "# Ensure dropout layers are in train mode\n",
        "encoder.train()\n",
        "decoder.train()\n",
        "\n",
        "# Initialize optimizers\n",
        "print('Building optimizers ...')\n",
        "encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate)\n",
        "decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate * decoder_learning_ratio)\n",
        "if loadFilename:\n",
        "    encoder_optimizer.load_state_dict(encoder_optimizer_sd)\n",
        "    decoder_optimizer.load_state_dict(decoder_optimizer_sd)\n",
        "\n",
        "# Run training iterations\n",
        "print(\"Starting Training!\")\n",
        "\n",
        "# Define the directory path in Google Drive\n",
        "save_dir = f'{FOLDER_PATH}/distilroberta_checkpoints'\n",
        "\n",
        "# Check if the directory exists\n",
        "if not os.path.exists(save_dir):\n",
        "    os.makedirs(save_dir)\n",
        "    print(f\"Directory '{save_dir}' created.\")\n",
        "else:\n",
        "    print(f\"Directory '{save_dir}' already exists.\")\n",
        "\n",
        "trainIters(model_name, encoder, decoder, encoder_optimizer, decoder_optimizer,\n",
        "           embedding, decoder_n_layers, save_dir, n_iteration, batch_size,\n",
        "           print_every, save_every, clip, loadFilename)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Initialization of Evaluation Function & GreedySearchDecoder**"
      ],
      "metadata": {
        "id": "fw_4Zpxf1sXF"
      },
      "id": "fw_4Zpxf1sXF"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "484e6fdc",
      "metadata": {
        "id": "484e6fdc"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer\n",
        "import torch\n",
        "import string\n",
        "import torch.nn as nn\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained('distilroberta-base')\n",
        "\n",
        "def preprocess_sentence(sentence):\n",
        "    sentence = sentence.lower()\n",
        "    sentence = sentence.translate(str.maketrans('', '', string.punctuation)).strip()\n",
        "    sentence = ' '.join(sentence.split())\n",
        "    return sentence\n",
        "\n",
        "def clean_output(decoded_words):\n",
        "    response = tokenizer.convert_tokens_to_string(decoded_words)\n",
        "    # Clean response\n",
        "    response = response.replace('Ġ', ' ').strip()\n",
        "    response = ' '.join(response.split())\n",
        "    return response\n",
        "\n",
        "def evaluate(encoder, decoder, searcher, sentence, max_length=MAX_LENGTH):\n",
        "    # Preprocess the input sentence\n",
        "    sentence = preprocess_sentence(sentence)\n",
        "\n",
        "    # Tokenize and encode the input sentence\n",
        "    inputs = tokenizer(sentence, return_tensors='pt', max_length=max_length, truncation=True, padding=True)\n",
        "    input_ids = inputs['input_ids'].to(device)\n",
        "    attention_mask = inputs['attention_mask'].to(device)\n",
        "\n",
        "    # Decode sentence with searcher\n",
        "    tokens, scores = searcher(input_ids, attention_mask, max_length)\n",
        "    tokens = tokens.view(-1)\n",
        "\n",
        "    # Convert token IDs to words\n",
        "    decoded_words = tokenizer.convert_ids_to_tokens(tokens.tolist(), skip_special_tokens=True)\n",
        "    cleaned_response = clean_output(decoded_words)\n",
        "    return cleaned_response\n",
        "\n",
        "def evaluateInput(encoder, decoder, searcher):\n",
        "    input_sentence = ''\n",
        "    while True:\n",
        "        try:\n",
        "            # Get input sentence\n",
        "            input_sentence = input('> ')\n",
        "            # Check if it is quit case\n",
        "            if input_sentence.lower() in ['q', 'quit']:\n",
        "                break\n",
        "\n",
        "            # Evaluate sentence\n",
        "            output_words = evaluate(encoder, decoder, searcher, input_sentence)\n",
        "\n",
        "            # Print response sentence\n",
        "            print('Bot:', output_words)\n",
        "\n",
        "        except KeyError:\n",
        "            print(\"Error: Encountered unknown word.\")\n",
        "\n",
        "class GreedySearchDecoder(nn.Module):\n",
        "    def __init__(self, encoder, decoder, device, tokenizer):\n",
        "        super(GreedySearchDecoder, self).__init__()\n",
        "        self.encoder = encoder\n",
        "        self.decoder = decoder\n",
        "        self._device = device\n",
        "        self.tokenizer = tokenizer\n",
        "\n",
        "    def forward(self, input_variable, attention_mask, max_length=MAX_LENGTH):\n",
        "        # Forward input through encoder model\n",
        "        encoder_outputs, encoder_last_hidden = self.encoder(input_ids=input_variable, attention_mask=attention_mask)\n",
        "\n",
        "        # Prepare encoder's final hidden layer to be first hidden input to the decoder\n",
        "        decoder_hidden = encoder_last_hidden\n",
        "\n",
        "        batch_size = input_variable.size(0)\n",
        "        SOS_token = \"<s>\"\n",
        "        SOS_token_id = self.tokenizer.convert_tokens_to_ids(SOS_token)\n",
        "\n",
        "        # Initialize decoder input with SOS token\n",
        "        decoder_input = torch.tensor([[SOS_token_id]] * batch_size).to(self._device)\n",
        "\n",
        "        # Initialize tensors to append decoded words to\n",
        "        all_tokens = torch.zeros([0], device=self._device, dtype=torch.long)\n",
        "        all_scores = torch.zeros([0], device=self._device)\n",
        "\n",
        "        # Iteratively decode one word token at a time\n",
        "        for _ in range(max_length):\n",
        "            # Forward pass through decoder\n",
        "            decoder_output, decoder_hidden = self.decoder(decoder_input, decoder_hidden, encoder_outputs)\n",
        "\n",
        "            # Obtain most likely word token and its softmax score\n",
        "            decoder_scores, decoder_input = torch.max(decoder_output, dim=1)\n",
        "\n",
        "            # Record token and score\n",
        "            all_tokens = torch.cat((all_tokens, decoder_input.unsqueeze(0)), dim=0)\n",
        "            all_scores = torch.cat((all_scores, decoder_scores.unsqueeze(0)), dim=0)\n",
        "\n",
        "            # Prepare current token to be next decoder input (add a dimension)\n",
        "            decoder_input = decoder_input.unsqueeze(0)\n",
        "\n",
        "        # Return collections of word tokens and scores\n",
        "        return all_tokens, all_scores"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Begin Chatting with the Chatbot here!**"
      ],
      "metadata": {
        "id": "oddsMqpK2Jh1"
      },
      "id": "oddsMqpK2Jh1"
    },
    {
      "cell_type": "code",
      "source": [
        "encoder.eval()\n",
        "decoder.eval()\n",
        "\n",
        "# Initialize search module\n",
        "searcher = GreedySearchDecoder(encoder, decoder, device, tokenizer)\n",
        "\n",
        "# Begin chatting\n",
        "evaluateInput(encoder, decoder, searcher)"
      ],
      "metadata": {
        "id": "aoCEquTs5nsX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 710
        },
        "outputId": "f143db7b-03b9-4a60-c6b0-015ac57d75fe"
      },
      "id": "aoCEquTs5nsX",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "> what is recursion\n",
            "Bot: recursion is a programming technique where a function calls itself in its own definition it allows for solving complex problems by breaking them down into smaller simpler subproblems that are solved recursively recursion can be used to solve problems that exhibit a divide and conquer or topdown approach where a problem is divided into smaller subproblems until a base case is reached recursion can be powerful but should be used with caution to prevent infinite loops or stack overflow errors\n",
            "> Explain the concept of recursion to me\n",
            "Bot: recursion is a programming technique where a function calls itself in its own definition it allows for solving complex problems by breaking them down into smaller simpler subproblems that are solved recursively recursion can be used to solve problems that exhibit a divide and conquer or topdown approach where a problem is divided into smaller subproblems until a base case is reached recursion can be powerful but should be used with caution to prevent infinite loops or stack overflow errors\n",
            "> define a kernel\n",
            "Bot: kernel is the part of os which handles all details of sharing resources and device handlingit can be considered as the core of os which manages the core features of an osits purpose is to handle the communication between software and hardwareits services are used through system calls and a layer of software called shell wraps around the kernel\n",
            "> Can you explain what is a kernel?\n",
            "Bot: kernel is the core and essential part of computer operating system that provides basic services for all parts of os\n",
            "> explain in brief about data imputation\n",
            "Bot: data science is a multidisciplinary field that involves extracting insights and knowledge from data using various techniques and tools the data science process typically includes steps such as data collection data cleaning data exploration data analysis and data visualization den visualization analysis in multivariate analysis\n",
            "> explain in brief about data imputation\n",
            "Bot: data science is a multidisciplinary field that involves extracting insights and knowledge from data using various techniques and tools the data science process typically includes steps such as data collection data cleaning data exploration data analysis and data visualization den visualization analysis in multivariate analysis\n",
            "> how do you explain a priority queue\n",
            "Bot: the priority queue is a data structure in which the intrinsic ordering of the elements numeric or alphabetic determines the result of its basic operation it is of two typesthey are ascending priority queue here smallest item can be removed insertion is arbitrarydescending priority queue here largest item can be removed insertion is arbitrary\n",
            "> what are the disadvantages array implementations of linked list\n",
            "Bot: the heterogeneous linked list contains different data types in its nodes and we need a link pointer to connect them it is not possible to use ordinary pointers for this so we go for void pointer void pointer is capable of storing pointer to any type as it is a generic pointer type\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "Interrupted by user",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-28-694d4a42cd06>\u001b[0m in \u001b[0;36m<cell line: 8>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m# Begin chatting\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0mevaluateInput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mencoder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecoder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msearcher\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-27-ebcebfc9d923>\u001b[0m in \u001b[0;36mevaluateInput\u001b[0;34m(encoder, decoder, searcher)\u001b[0m\n\u001b[1;32m     42\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     43\u001b[0m             \u001b[0;31m# Get input sentence\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 44\u001b[0;31m             \u001b[0minput_sentence\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'> '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     45\u001b[0m             \u001b[0;31m# Check if it is quit case\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     46\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0minput_sentence\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'q'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'quit'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py\u001b[0m in \u001b[0;36mraw_input\u001b[0;34m(self, prompt)\u001b[0m\n\u001b[1;32m    849\u001b[0m                 \u001b[0;34m\"raw_input was called, but this frontend does not support input requests.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    850\u001b[0m             )\n\u001b[0;32m--> 851\u001b[0;31m         return self._input_request(str(prompt),\n\u001b[0m\u001b[1;32m    852\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parent_ident\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    853\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parent_header\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py\u001b[0m in \u001b[0;36m_input_request\u001b[0;34m(self, prompt, ident, parent, password)\u001b[0m\n\u001b[1;32m    893\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mKeyboardInterrupt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m                 \u001b[0;31m# re-raise KeyboardInterrupt, to truncate traceback\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 895\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mKeyboardInterrupt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Interrupted by user\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    896\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlog\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarning\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Invalid Message:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexc_info\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: Interrupted by user"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### BERTScore evaluation"
      ],
      "metadata": {
        "id": "QiNsbbyrW-xR"
      },
      "id": "QiNsbbyrW-xR"
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install bert_score"
      ],
      "metadata": {
        "id": "ZkuFxPrfW-Lx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5c6d070d-f907-45c1-c70a-71e8f1cbe97f"
      },
      "id": "ZkuFxPrfW-Lx",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting bert_score\n",
            "  Downloading bert_score-0.3.13-py3-none-any.whl.metadata (15 kB)\n",
            "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from bert_score) (2.5.1+cu121)\n",
            "Requirement already satisfied: pandas>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from bert_score) (2.2.2)\n",
            "Requirement already satisfied: transformers>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from bert_score) (4.46.2)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from bert_score) (1.26.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from bert_score) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.31.1 in /usr/local/lib/python3.10/dist-packages (from bert_score) (4.66.6)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from bert_score) (3.8.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from bert_score) (24.2)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert_score) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert_score) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert_score) (2024.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (3.16.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (4.12.2)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (2024.10.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert_score) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch>=1.0.0->bert_score) (1.3.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (0.26.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (6.0.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (2024.9.11)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (0.4.5)\n",
            "Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert_score) (0.20.3)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (4.55.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (1.4.7)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (11.0.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert_score) (3.2.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->bert_score) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->bert_score) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->bert_score) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->bert_score) (2024.8.30)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas>=1.0.1->bert_score) (1.16.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.0.0->bert_score) (3.0.2)\n",
            "Downloading bert_score-0.3.13-py3-none-any.whl (61 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.1/61.1 kB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: bert_score\n",
            "Successfully installed bert_score-0.3.13\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from bert_score import score\n",
        "\n",
        "# Generate responses for all questions\n",
        "encoder.eval()\n",
        "decoder.eval()\n",
        "generated_responses = []\n",
        "\n",
        "with torch.no_grad():\n",
        "    for question in questions:\n",
        "        response = evaluate(encoder, decoder, searcher, question)\n",
        "        generated_responses.append(response)  # Store the generated response\n",
        "\n",
        "# Step 4: Compute BERTScore\n",
        "P, R, F1 = score(generated_responses, answers, model_type='distilroberta-base', verbose=True)\n",
        "\n",
        "# Prepare data for CSV\n",
        "results = []\n",
        "for idx, (p, r, f1) in enumerate(zip(P, R, F1)):\n",
        "    results.append({\n",
        "        \"Question\": questions[idx],\n",
        "        \"Ground Truth\": answers[idx],\n",
        "        \"Generated Response\": generated_responses[idx],\n",
        "        \"Precision\": p.item(),\n",
        "        \"Recall\": r.item(),\n",
        "        \"F1\": f1.item()\n",
        "    })\n",
        "\n",
        "# Create a DataFrame\n",
        "results_df = pd.DataFrame(results)\n",
        "\n",
        "# Save the DataFrame to a CSV file\n",
        "results_df.to_csv(f'{FOLDER_PATH}/DistilRoberta_bert_scores.csv', index=False)\n",
        "\n",
        "# Calculate average BERTScores\n",
        "avg_precision = results_df[\"Precision\"].mean()\n",
        "avg_recall = results_df[\"Recall\"].mean()\n",
        "avg_f1 = results_df[\"F1\"].mean()\n",
        "\n",
        "# Print averages\n",
        "print(f\"Average Precision: {avg_precision:.4f}\")\n",
        "print(f\"Average Recall: {avg_recall:.4f}\")\n",
        "print(f\"Average F1 Score: {avg_f1:.4f}\")"
      ],
      "metadata": {
        "id": "wCjM5x6ZYkgo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 200,
          "referenced_widgets": [
            "5d9090a83b5347e48f6f1cdf2df5a328",
            "3a6071f3296b4849bf003d687df415a1",
            "b34df29b948d4734a5a3975159d2cd51",
            "b2a11171f0c2400aba04ae818d900c71",
            "d7205db957ae401590f6f20b982a0679",
            "3d3c4d30568d4779a1959ce7d9790f06",
            "918abbb2446b411b91ceb09d166bfbe1",
            "4243042ff99d4d8c9299d01956c7ac4d",
            "f8a8b432bdfa44bf8f25f99e587dd703",
            "d446ee37d8cf4720af7eaca72ed37940",
            "217f6097c4004095b7b4a882011bd784",
            "4243b9adfc5349929a93b9bec08bb830",
            "aabdb4bd2c2d42e49ed6d6cb61d23665",
            "f96557b8bf2946f1b97b128a9c410a65",
            "1364437abfe74854a18cefc685442995",
            "dec5b09c38f24c0e80d860edfcde997f",
            "1ea5f0f368224b99960157ea52e24269",
            "ca9b527198d6412ba0f891d9d0b34623",
            "5f8c812acd774bbcba9e0ab5b17b4337",
            "b2793eb078c444c7ba91fe82405ce79b",
            "c14c446f38244d818d785ccb1e194b14",
            "3c9efb5d55ba4012afafc7f3a0905296"
          ]
        },
        "outputId": "1e9d2bea-760d-4c4c-af1a-855a320fdc0f"
      },
      "id": "wCjM5x6ZYkgo",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "calculating scores...\n",
            "computing bert embedding.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/40 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "5d9090a83b5347e48f6f1cdf2df5a328"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "computing greedy matching.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/59 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "4243b9adfc5349929a93b9bec08bb830"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "done in 5.21 seconds, 725.08 sentences/sec\n",
            "Average Precision: 0.9705\n",
            "Average Recall: 0.9705\n",
            "Average F1 Score: 0.9704\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install nltk\n",
        "!pip install bert-score\n",
        "!pip install rouge-score"
      ],
      "metadata": {
        "id": "sUf9oHU1vZnw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eeac88cf-4a4c-4ec7-d7d5-b80fc3bbd855"
      },
      "id": "sUf9oHU1vZnw",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (3.9.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk) (8.1.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk) (1.4.2)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk) (2024.9.11)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nltk) (4.66.6)\n",
            "Requirement already satisfied: bert-score in /usr/local/lib/python3.10/dist-packages (0.3.13)\n",
            "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from bert-score) (2.5.1+cu121)\n",
            "Requirement already satisfied: pandas>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from bert-score) (2.2.2)\n",
            "Requirement already satisfied: transformers>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from bert-score) (4.46.2)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from bert-score) (1.26.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from bert-score) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.31.1 in /usr/local/lib/python3.10/dist-packages (from bert-score) (4.66.6)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from bert-score) (3.8.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from bert-score) (24.2)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert-score) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert-score) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert-score) (2024.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score) (3.16.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score) (4.12.2)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score) (2024.10.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch>=1.0.0->bert-score) (1.3.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert-score) (0.26.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert-score) (6.0.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert-score) (2024.9.11)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert-score) (0.4.5)\n",
            "Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers>=3.0.0->bert-score) (0.20.3)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score) (4.55.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score) (1.4.7)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score) (11.0.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score) (3.2.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score) (2024.8.30)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas>=1.0.1->bert-score) (1.16.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.0.0->bert-score) (3.0.2)\n",
            "Collecting rouge-score\n",
            "  Downloading rouge_score-0.1.2.tar.gz (17 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from rouge-score) (1.4.0)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score) (3.9.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from rouge-score) (1.26.4)\n",
            "Requirement already satisfied: six>=1.14.0 in /usr/local/lib/python3.10/dist-packages (from rouge-score) (1.16.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->rouge-score) (8.1.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk->rouge-score) (1.4.2)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk->rouge-score) (2024.9.11)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nltk->rouge-score) (4.66.6)\n",
            "Building wheels for collected packages: rouge-score\n",
            "  Building wheel for rouge-score (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24935 sha256=6396f6a8ab8e3c5d1c8d9820a807e16208b4f3f2b433431d687b1e724912a141\n",
            "  Stored in directory: /root/.cache/pip/wheels/5f/dd/89/461065a73be61a532ff8599a28e9beef17985c9e9c31e541b4\n",
            "Successfully built rouge-score\n",
            "Installing collected packages: rouge-score\n",
            "Successfully installed rouge-score-0.1.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('punkt')"
      ],
      "metadata": {
        "id": "3ZcLBNJXybyu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31a11029-54df-4479-d439-6c6c95b8800a"
      },
      "id": "3ZcLBNJXybyu",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from bert_score import score\n",
        "import nltk\n",
        "from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction\n",
        "from rouge_score import rouge_scorer\n",
        "\n",
        "# Generate responses for all questions\n",
        "encoder.eval()\n",
        "decoder.eval()\n",
        "generated_responses = []\n",
        "\n",
        "with torch.no_grad():\n",
        "    for question in questions:\n",
        "        response = evaluate(encoder, decoder, searcher, question)\n",
        "        generated_responses.append(response)  # Store the generated response\n",
        "\n",
        "# Compute BERTScore\n",
        "P, R, F1 = score(\n",
        "    generated_responses,\n",
        "    answers,\n",
        "    model_type='distilroberta-base',\n",
        "    verbose=True\n",
        ")\n",
        "\n",
        "# Initialize BLEU smoothing function and ROUGE scorer\n",
        "from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction\n",
        "smoothie = SmoothingFunction().method4\n",
        "rouge = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)\n",
        "\n",
        "# Prepare data for CSV\n",
        "results = []\n",
        "for idx, (question, reference, candidate, p, r, f1) in enumerate(zip(questions, answers, generated_responses, P, R, F1)):\n",
        "    # Tokenize using transformers' tokenizer\n",
        "    reference_tokens = tokenizer.tokenize(reference.lower())\n",
        "    candidate_tokens = tokenizer.tokenize(candidate.lower())\n",
        "\n",
        "    # Compute BLEU score\n",
        "    bleu = sentence_bleu([reference_tokens], candidate_tokens, smoothing_function=smoothie)\n",
        "\n",
        "    # Compute ROUGE scores (ROUGE expects strings, no tokenization needed)\n",
        "    scores = rouge.score(reference.lower(), candidate.lower())\n",
        "    rouge1_f1 = scores['rouge1'].fmeasure\n",
        "    rougeL_f1 = scores['rougeL'].fmeasure\n",
        "\n",
        "    # Store the results\n",
        "    results.append({\n",
        "        \"Question\": question,\n",
        "        \"Ground Truth\": reference,\n",
        "        \"Generated Response\": candidate,\n",
        "        \"Precision\": p.item(),\n",
        "        \"Recall\": r.item(),\n",
        "        \"F1\": f1.item(),\n",
        "        \"BLEU\": bleu,\n",
        "        \"ROUGE-1 F1\": rouge1_f1,\n",
        "        \"ROUGE-L F1\": rougeL_f1\n",
        "    })\n",
        "\n",
        "# Create a DataFrame\n",
        "results_df = pd.DataFrame(results)\n",
        "\n",
        "# Save the DataFrame to a CSV file\n",
        "results_df.to_csv(f'{FOLDER_PATH}/bryan_model/DistilRoberta_bert_scores.csv', index=False)\n",
        "\n",
        "# Calculate average BERTScores\n",
        "avg_precision = results_df[\"Precision\"].mean()\n",
        "avg_recall = results_df[\"Recall\"].mean()\n",
        "avg_f1 = results_df[\"F1\"].mean()\n",
        "\n",
        "# Calculate average BLEU and ROUGE scores\n",
        "avg_bleu = results_df[\"BLEU\"].mean()\n",
        "avg_rouge1_f1 = results_df[\"ROUGE-1 F1\"].mean()\n",
        "avg_rougeL_f1 = results_df[\"ROUGE-L F1\"].mean()\n",
        "\n",
        "# Print averages\n",
        "print(f\"Average BERTScore Precision: {avg_precision:.4f}\")\n",
        "print(f\"Average BERTScore Recall: {avg_recall:.4f}\")\n",
        "print(f\"Average BERTScore F1 Score: {avg_f1:.4f}\")\n",
        "print(f\"Average BLEU Score: {avg_bleu:.4f}\")\n",
        "print(f\"Average ROUGE-1 F1 Score: {avg_rouge1_f1:.4f}\")\n",
        "print(f\"Average ROUGE-L F1 Score: {avg_rougeL_f1:.4f}\")"
      ],
      "metadata": {
        "id": "p0m5PQxerpaA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 251,
          "referenced_widgets": [
            "350dec149da24806b9d2ba851cd481e2",
            "48d9a3085ad141bebf84c3b613afdbf1",
            "b79e10273c544a67939eb0d880fb4606",
            "72ff6ecb6aa940e9a8ac0a796bae2a38",
            "07f77482cbc947978558d330eaf42e34",
            "307b5ec67e584e6a8d280e59ba2fdaa5",
            "c6fa50963033464dbc9c3e40953754c9",
            "81999149115f4251851922c33575055a",
            "b257f126175c44d5928352f1707641bd",
            "91b2637f42fe4e488cf322d16b61a9c2",
            "bc6ca21e6eab45828be5d3079706db37",
            "147af560c7f1486bb8e707186e4f05de",
            "41a47922072c4ac3a42e05397fb00d6e",
            "e86d5cd71ed44ef8aa34b280212f5dc8",
            "90aabed1ba0e4ef5b84d9485188d7802",
            "bd47e64592f74b638dd22d4ac8942b6a",
            "e8de99eeef2e45fd95430a72e4d2b494",
            "401aa626425c44bca70617a91e1617ea",
            "855c21871b534c63b50ad94836b8bb28",
            "45106f9630dc4f22bb85e428f763213e",
            "7e17520334ae45a4a42dd87d7f713692",
            "c9956ec008f940f399e0e6849494aab7"
          ]
        },
        "outputId": "e8bd39d3-557a-4877-a1f8-f34e9a5c4584"
      },
      "id": "p0m5PQxerpaA",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "calculating scores...\n",
            "computing bert embedding.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/40 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "350dec149da24806b9d2ba851cd481e2"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "computing greedy matching.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/59 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "147af560c7f1486bb8e707186e4f05de"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "done in 5.48 seconds, 688.60 sentences/sec\n",
            "Average BERTScore Precision: 0.9705\n",
            "Average BERTScore Recall: 0.9705\n",
            "Average BERTScore F1 Score: 0.9704\n",
            "Average BLEU Score: 0.7080\n",
            "Average ROUGE-1 F1 Score: 0.7943\n",
            "Average ROUGE-L F1 Score: 0.7731\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Telegram Implementation\n",
        "- Replace token with your own API token obtained by BotFather on Telegram\n",
        "- application = Application.builder().token('**Replace token here**').build()\n",
        ""
      ],
      "metadata": {
        "id": "icPY3cbo3ksU"
      },
      "id": "icPY3cbo3ksU"
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install python-telegram-bot --upgrade\n",
        "!pip install nest_asyncio"
      ],
      "metadata": {
        "id": "_kqmgHsursMX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "51e40151-7c03-486a-981d-b7b6f16960f4"
      },
      "id": "_kqmgHsursMX",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting python-telegram-bot\n",
            "  Downloading python_telegram_bot-21.7-py3-none-any.whl.metadata (17 kB)\n",
            "Requirement already satisfied: httpx~=0.27 in /usr/local/lib/python3.10/dist-packages (from python-telegram-bot) (0.27.2)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (3.7.1)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (2024.8.30)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (1.0.7)\n",
            "Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (3.10)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx~=0.27->python-telegram-bot) (1.3.1)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx~=0.27->python-telegram-bot) (0.14.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx~=0.27->python-telegram-bot) (1.2.2)\n",
            "Downloading python_telegram_bot-21.7-py3-none-any.whl (654 kB)\n",
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/654.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━\u001b[0m \u001b[32m634.9/654.9 kB\u001b[0m \u001b[31m20.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m654.9/654.9 kB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: python-telegram-bot\n",
            "Successfully installed python-telegram-bot-21.7\n",
            "Requirement already satisfied: nest_asyncio in /usr/local/lib/python3.10/dist-packages (1.6.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import logging\n",
        "import asyncio\n",
        "import nest_asyncio\n",
        "from telegram import Update\n",
        "from telegram.ext import Application, MessageHandler, filters, ContextTypes\n",
        "\n",
        "# Apply the nest_asyncio patch for Jupyter environments\n",
        "nest_asyncio.apply()\n",
        "\n",
        "# Configure logging\n",
        "logging.basicConfig(\n",
        "    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',\n",
        "    level=logging.INFO\n",
        ")\n",
        "\n",
        "# Define the function to process user input and generate responses using your model\n",
        "async def respond(update: Update, context: ContextTypes.DEFAULT_TYPE):\n",
        "    user_message = update.message.text\n",
        "    # Preprocess user message if necessary\n",
        "    input_sentence = user_message  # Or use preprocess_sentence(user_message)\n",
        "\n",
        "    # Generate a response using your evaluate function\n",
        "    response = evaluate(encoder, decoder, searcher, input_sentence)\n",
        "\n",
        "    # Send the response back to the user\n",
        "    await update.message.reply_text(response)\n",
        "\n",
        "# Main function to set up and run the bot\n",
        "async def main():\n",
        "    # Replace 'YOUR_TOKEN' with your bot's API token\n",
        "    application = Application.builder().token('Replace token here').build()\n",
        "\n",
        "    # Add handler for text messages\n",
        "    application.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, respond))\n",
        "\n",
        "    # Initialize, start, and poll the bot\n",
        "    await application.initialize()\n",
        "    await application.start()\n",
        "    await application.updater.start_polling()\n",
        "    # Keep the bot running\n",
        "    await asyncio.Event().wait()\n",
        "\n",
        "# Run the bot using asyncio\n",
        "asyncio.run(main())"
      ],
      "metadata": {
        "id": "ARGKGxLa3qmT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 367
        },
        "outputId": "1a5038ac-b2be-48ba-9970-bc9c5d13e416"
      },
      "id": "ARGKGxLa3qmT",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-37-01cd50b9cb5e>\u001b[0m in \u001b[0;36m<cell line: 44>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     42\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     43\u001b[0m \u001b[0;31m# Run the bot using asyncio\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 44\u001b[0;31m \u001b[0masyncio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/nest_asyncio.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(main, debug)\u001b[0m\n\u001b[1;32m     28\u001b[0m         \u001b[0mtask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0masyncio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_future\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmain\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 30\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mloop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_until_complete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtask\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     31\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     32\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mtask\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/nest_asyncio.py\u001b[0m in \u001b[0;36mrun_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m     90\u001b[0m                 \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_log_destroy_pending\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m             \u001b[0;32mwhile\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 92\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_once\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     93\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stopping\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     94\u001b[0m                     \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/nest_asyncio.py\u001b[0m in \u001b[0;36m_run_once\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    113\u001b[0m                 scheduled[0]._when - self.time(), 0), 86400) if scheduled\n\u001b[1;32m    114\u001b[0m             else None)\n\u001b[0;32m--> 115\u001b[0;31m         \u001b[0mevent_list\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_selector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mselect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    116\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_process_events\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mevent_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    117\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.10/selectors.py\u001b[0m in \u001b[0;36mselect\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    467\u001b[0m             \u001b[0mready\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    468\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 469\u001b[0;31m                 \u001b[0mfd_event_list\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_selector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpoll\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_ev\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    470\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mInterruptedError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    471\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mready\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.5"
    },
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "5d9090a83b5347e48f6f1cdf2df5a328": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3a6071f3296b4849bf003d687df415a1",
              "IPY_MODEL_b34df29b948d4734a5a3975159d2cd51",
              "IPY_MODEL_b2a11171f0c2400aba04ae818d900c71"
            ],
            "layout": "IPY_MODEL_d7205db957ae401590f6f20b982a0679"
          }
        },
        "3a6071f3296b4849bf003d687df415a1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3d3c4d30568d4779a1959ce7d9790f06",
            "placeholder": "​",
            "style": "IPY_MODEL_918abbb2446b411b91ceb09d166bfbe1",
            "value": "100%"
          }
        },
        "b34df29b948d4734a5a3975159d2cd51": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4243042ff99d4d8c9299d01956c7ac4d",
            "max": 40,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_f8a8b432bdfa44bf8f25f99e587dd703",
            "value": 40
          }
        },
        "b2a11171f0c2400aba04ae818d900c71": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d446ee37d8cf4720af7eaca72ed37940",
            "placeholder": "​",
            "style": "IPY_MODEL_217f6097c4004095b7b4a882011bd784",
            "value": " 40/40 [00:04&lt;00:00, 17.82it/s]"
          }
        },
        "d7205db957ae401590f6f20b982a0679": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3d3c4d30568d4779a1959ce7d9790f06": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "918abbb2446b411b91ceb09d166bfbe1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4243042ff99d4d8c9299d01956c7ac4d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f8a8b432bdfa44bf8f25f99e587dd703": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d446ee37d8cf4720af7eaca72ed37940": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "217f6097c4004095b7b4a882011bd784": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4243b9adfc5349929a93b9bec08bb830": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_aabdb4bd2c2d42e49ed6d6cb61d23665",
              "IPY_MODEL_f96557b8bf2946f1b97b128a9c410a65",
              "IPY_MODEL_1364437abfe74854a18cefc685442995"
            ],
            "layout": "IPY_MODEL_dec5b09c38f24c0e80d860edfcde997f"
          }
        },
        "aabdb4bd2c2d42e49ed6d6cb61d23665": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1ea5f0f368224b99960157ea52e24269",
            "placeholder": "​",
            "style": "IPY_MODEL_ca9b527198d6412ba0f891d9d0b34623",
            "value": "100%"
          }
        },
        "f96557b8bf2946f1b97b128a9c410a65": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5f8c812acd774bbcba9e0ab5b17b4337",
            "max": 59,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b2793eb078c444c7ba91fe82405ce79b",
            "value": 59
          }
        },
        "1364437abfe74854a18cefc685442995": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c14c446f38244d818d785ccb1e194b14",
            "placeholder": "​",
            "style": "IPY_MODEL_3c9efb5d55ba4012afafc7f3a0905296",
            "value": " 59/59 [00:00&lt;00:00, 65.23it/s]"
          }
        },
        "dec5b09c38f24c0e80d860edfcde997f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1ea5f0f368224b99960157ea52e24269": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ca9b527198d6412ba0f891d9d0b34623": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5f8c812acd774bbcba9e0ab5b17b4337": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b2793eb078c444c7ba91fe82405ce79b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c14c446f38244d818d785ccb1e194b14": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3c9efb5d55ba4012afafc7f3a0905296": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "350dec149da24806b9d2ba851cd481e2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_48d9a3085ad141bebf84c3b613afdbf1",
              "IPY_MODEL_b79e10273c544a67939eb0d880fb4606",
              "IPY_MODEL_72ff6ecb6aa940e9a8ac0a796bae2a38"
            ],
            "layout": "IPY_MODEL_07f77482cbc947978558d330eaf42e34"
          }
        },
        "48d9a3085ad141bebf84c3b613afdbf1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_307b5ec67e584e6a8d280e59ba2fdaa5",
            "placeholder": "​",
            "style": "IPY_MODEL_c6fa50963033464dbc9c3e40953754c9",
            "value": "100%"
          }
        },
        "b79e10273c544a67939eb0d880fb4606": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_81999149115f4251851922c33575055a",
            "max": 40,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b257f126175c44d5928352f1707641bd",
            "value": 40
          }
        },
        "72ff6ecb6aa940e9a8ac0a796bae2a38": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_91b2637f42fe4e488cf322d16b61a9c2",
            "placeholder": "​",
            "style": "IPY_MODEL_bc6ca21e6eab45828be5d3079706db37",
            "value": " 40/40 [00:04&lt;00:00, 13.98it/s]"
          }
        },
        "07f77482cbc947978558d330eaf42e34": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "307b5ec67e584e6a8d280e59ba2fdaa5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c6fa50963033464dbc9c3e40953754c9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "81999149115f4251851922c33575055a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b257f126175c44d5928352f1707641bd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "91b2637f42fe4e488cf322d16b61a9c2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bc6ca21e6eab45828be5d3079706db37": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "147af560c7f1486bb8e707186e4f05de": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_41a47922072c4ac3a42e05397fb00d6e",
              "IPY_MODEL_e86d5cd71ed44ef8aa34b280212f5dc8",
              "IPY_MODEL_90aabed1ba0e4ef5b84d9485188d7802"
            ],
            "layout": "IPY_MODEL_bd47e64592f74b638dd22d4ac8942b6a"
          }
        },
        "41a47922072c4ac3a42e05397fb00d6e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e8de99eeef2e45fd95430a72e4d2b494",
            "placeholder": "​",
            "style": "IPY_MODEL_401aa626425c44bca70617a91e1617ea",
            "value": "100%"
          }
        },
        "e86d5cd71ed44ef8aa34b280212f5dc8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_855c21871b534c63b50ad94836b8bb28",
            "max": 59,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_45106f9630dc4f22bb85e428f763213e",
            "value": 59
          }
        },
        "90aabed1ba0e4ef5b84d9485188d7802": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7e17520334ae45a4a42dd87d7f713692",
            "placeholder": "​",
            "style": "IPY_MODEL_c9956ec008f940f399e0e6849494aab7",
            "value": " 59/59 [00:00&lt;00:00, 64.70it/s]"
          }
        },
        "bd47e64592f74b638dd22d4ac8942b6a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e8de99eeef2e45fd95430a72e4d2b494": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "401aa626425c44bca70617a91e1617ea": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "855c21871b534c63b50ad94836b8bb28": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "45106f9630dc4f22bb85e428f763213e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7e17520334ae45a4a42dd87d7f713692": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c9956ec008f940f399e0e6849494aab7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "be531bb8ef274e34925835b254756cc2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_02ca0cf92e5c420d8b9d9214d18eec0c",
              "IPY_MODEL_201922e1c2e2419dbe1473c192d7f864",
              "IPY_MODEL_831c6f9492d9491fa07c8d14263b23c6"
            ],
            "layout": "IPY_MODEL_2d53b8369c7c4d25995b45e7df2f39ac"
          }
        },
        "02ca0cf92e5c420d8b9d9214d18eec0c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cb5d5a61ac484e62ac862ae2e8c4d35b",
            "placeholder": "​",
            "style": "IPY_MODEL_5bc766a4b59640c4ad4175f44b3c6114",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "201922e1c2e2419dbe1473c192d7f864": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_91e9160c803740d183082fbbbd285437",
            "max": 25,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_348c563c16a648d68569a967c7dd0d20",
            "value": 25
          }
        },
        "831c6f9492d9491fa07c8d14263b23c6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_56d8b599a7dd4c0dbf00483ab7d06f39",
            "placeholder": "​",
            "style": "IPY_MODEL_c07be4243736443bad0090f05f7df2e1",
            "value": " 25.0/25.0 [00:00&lt;00:00, 1.90kB/s]"
          }
        },
        "2d53b8369c7c4d25995b45e7df2f39ac": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cb5d5a61ac484e62ac862ae2e8c4d35b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5bc766a4b59640c4ad4175f44b3c6114": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "91e9160c803740d183082fbbbd285437": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "348c563c16a648d68569a967c7dd0d20": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "56d8b599a7dd4c0dbf00483ab7d06f39": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c07be4243736443bad0090f05f7df2e1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7ac28328549d428aab0c715116d3579f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a1c3b002f9a543dfbc05dfa880fc934f",
              "IPY_MODEL_b2ff67d69ba142bb95e7d4c8388a6935",
              "IPY_MODEL_2be8dde9528445fbae429a25988f228d"
            ],
            "layout": "IPY_MODEL_c08fe52e280d4cab903ab36f4a550238"
          }
        },
        "a1c3b002f9a543dfbc05dfa880fc934f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8ea51d2da3a345a49990d5cb9839c073",
            "placeholder": "​",
            "style": "IPY_MODEL_e390578f1e6f4e04aabadb35b03ea5d8",
            "value": "config.json: 100%"
          }
        },
        "b2ff67d69ba142bb95e7d4c8388a6935": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2668a12bb7ad49b8824c6c3240e82ce5",
            "max": 480,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1c7e94e5c08943788c1fdf1a5c40749b",
            "value": 480
          }
        },
        "2be8dde9528445fbae429a25988f228d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ea5a79e74de741cca7f0af1fe7cc2f1f",
            "placeholder": "​",
            "style": "IPY_MODEL_ccfac9b66fea4c4a835a03f809aeca6a",
            "value": " 480/480 [00:00&lt;00:00, 31.1kB/s]"
          }
        },
        "c08fe52e280d4cab903ab36f4a550238": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8ea51d2da3a345a49990d5cb9839c073": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e390578f1e6f4e04aabadb35b03ea5d8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2668a12bb7ad49b8824c6c3240e82ce5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1c7e94e5c08943788c1fdf1a5c40749b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ea5a79e74de741cca7f0af1fe7cc2f1f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ccfac9b66fea4c4a835a03f809aeca6a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0c6ee3ce8a68494c82298201606e3786": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_adb966239e61491791998da41c830d30",
              "IPY_MODEL_99b3437b804e45808a21ab5c8c6fd84d",
              "IPY_MODEL_c092685faef545f2837a2230495e4e1c"
            ],
            "layout": "IPY_MODEL_9bfeb9640bde4c57b082e1007fec6d4e"
          }
        },
        "adb966239e61491791998da41c830d30": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f1780a56fc5640279e9a31b4f909d152",
            "placeholder": "​",
            "style": "IPY_MODEL_5f14cea471404f5da11243f097020ede",
            "value": "vocab.json: 100%"
          }
        },
        "99b3437b804e45808a21ab5c8c6fd84d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1e54eb3786ec44729ed29a06ff8e402a",
            "max": 898823,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_90a6ec622d8d4106abc3e1f7b224675d",
            "value": 898823
          }
        },
        "c092685faef545f2837a2230495e4e1c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_568e2f77748241b6811ef2ae6196f116",
            "placeholder": "​",
            "style": "IPY_MODEL_d71571cc227248729352a1ac258adc30",
            "value": " 899k/899k [00:00&lt;00:00, 3.65MB/s]"
          }
        },
        "9bfeb9640bde4c57b082e1007fec6d4e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f1780a56fc5640279e9a31b4f909d152": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5f14cea471404f5da11243f097020ede": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1e54eb3786ec44729ed29a06ff8e402a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "90a6ec622d8d4106abc3e1f7b224675d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "568e2f77748241b6811ef2ae6196f116": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d71571cc227248729352a1ac258adc30": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "73e9503f09204413a5d4a511e7d8822f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ae259abd8a7b404296b9c6df42513d22",
              "IPY_MODEL_663a6ca2b1404a48a1510e42429a4d17",
              "IPY_MODEL_89726182aeac446eb7a52d6d423432ec"
            ],
            "layout": "IPY_MODEL_39305bc8add642b5866fe4dd93581762"
          }
        },
        "ae259abd8a7b404296b9c6df42513d22": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c27490a375864c8b89fce811796249c0",
            "placeholder": "​",
            "style": "IPY_MODEL_a5273db64fa94e5b8c643094557bc9a2",
            "value": "merges.txt: 100%"
          }
        },
        "663a6ca2b1404a48a1510e42429a4d17": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a754a76d0a7e42d1ad7940f3c410ae25",
            "max": 456318,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9d452f9d7f4845c2bee5b0fd729b89f3",
            "value": 456318
          }
        },
        "89726182aeac446eb7a52d6d423432ec": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4631ae7f63fa41d3bb499dfdc532489a",
            "placeholder": "​",
            "style": "IPY_MODEL_21b6b90a90254ecd895cac314b611a4f",
            "value": " 456k/456k [00:00&lt;00:00, 2.73MB/s]"
          }
        },
        "39305bc8add642b5866fe4dd93581762": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c27490a375864c8b89fce811796249c0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a5273db64fa94e5b8c643094557bc9a2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a754a76d0a7e42d1ad7940f3c410ae25": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9d452f9d7f4845c2bee5b0fd729b89f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4631ae7f63fa41d3bb499dfdc532489a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "21b6b90a90254ecd895cac314b611a4f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c5178b2628ae4378b8659d7aae5e8b3f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_51c00a71b3554f8985b43abf988192a4",
              "IPY_MODEL_4a69ab073e03496f898310e1fdaeb43e",
              "IPY_MODEL_1662089c61324150b997413129eeae7f"
            ],
            "layout": "IPY_MODEL_bbfa7628c1434b08887a32bd445dacf0"
          }
        },
        "51c00a71b3554f8985b43abf988192a4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9b1d7d1c2ce743c09b5a2324c11ba7e1",
            "placeholder": "​",
            "style": "IPY_MODEL_dfae4c8fa454402faec1a824afeb4bc0",
            "value": "tokenizer.json: 100%"
          }
        },
        "4a69ab073e03496f898310e1fdaeb43e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_99c6857bce134261949392f4ca0c3ccb",
            "max": 1355863,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_036bf95bfcef4707bbb11639f1d633bb",
            "value": 1355863
          }
        },
        "1662089c61324150b997413129eeae7f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_353a55607f7f4a628d94024e01c000cc",
            "placeholder": "​",
            "style": "IPY_MODEL_2bd9dc5b40f44d10a2e8097a83147a3c",
            "value": " 1.36M/1.36M [00:00&lt;00:00, 8.15MB/s]"
          }
        },
        "bbfa7628c1434b08887a32bd445dacf0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9b1d7d1c2ce743c09b5a2324c11ba7e1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dfae4c8fa454402faec1a824afeb4bc0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "99c6857bce134261949392f4ca0c3ccb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "036bf95bfcef4707bbb11639f1d633bb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "353a55607f7f4a628d94024e01c000cc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2bd9dc5b40f44d10a2e8097a83147a3c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}