analysis_round1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DynaSent version 1: Round 1 analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = 'Christopher Potts'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Set-up](#Set-up)\n",
    "1. [Load the splits](#Load-the-splits)\n",
    "1. [Label distributions](#Label-distributions)\n",
    "1. [Relationship between validation labels and model predictions](#Relationship-between-validation-labels-and-model-predictions)\n",
    "1. [Adversarial rates](#Adversarial-rates)\n",
    "1. [Relationship between validation labels and review-level ratings](#Relationship-between-validation-labels-and-review-level-ratings)\n",
    "1. [Distribution of validation response distributions](#Distribution-of-validation-response-distributions)\n",
    "1. [Worker distribution](#Worker-distribution)\n",
    "1. [Estimates of human performance](#Estimates-of-human-performance)\n",
    "1. [Examples for the paper](#Examples-for-the-paper)\n",
    "1. [Random cases](#Random-cases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "\n",
    "import dynasent_utils as utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.style.use(\"dynasent.mplstyle\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "src_dirname = \"dynasent-v1.1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename_template = os.path.join(\n",
    "    src_dirname, \"dynasent-v1.1-round01-yelp-{}.jsonl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_filename = filename_template.format(\"train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_filename = filename_template.format(\"dev\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_filename = filename_template.format(\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = utils.load_dataset(train_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'hit_ids': ['y5238'],\n",
       " 'sentence': 'Roto-Rooter is always good when you need someone right away.',\n",
       " 'indices_into_review_text': [0, 60],\n",
       " 'model_0_label': 'positive',\n",
       " 'model_0_probs': {'negative': 0.01173639390617609,\n",
       "  'positive': 0.7473671436309814,\n",
       "  'neutral': 0.24089649319648743},\n",
       " 'text_id': 'IDHkeGo-nxhqX4Exkdr08A',\n",
       " 'review_rating': 1,\n",
       " 'label_distribution': {'positive': ['w130', 'w186', 'w207', 'w264', 'w54'],\n",
       "  'negative': [],\n",
       "  'neutral': [],\n",
       "  'mixed': []},\n",
       " 'gold_label': 'positive'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ternary = utils.load_dataset(\n",
    "    train_filename,\n",
    "    labels=('positive', 'negative', 'neutral'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev = utils.load_dataset(dev_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = utils.load_dataset(test_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "neutral         47476\n",
       "positive        23791\n",
       "negative        16421\n",
       "No Majority     10071\n",
       "mixed            3900\n",
       "Total          101659\n",
       "Name: gold_label, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_distribution(train, dev, test, dist_labels=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "neutral     215935\n",
       "positive    130045\n",
       "negative     86486\n",
       "mixed        39829\n",
       "Total       472295\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_distribution(train, dist_labels=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_cat = utils.get_label_distribution(train, dist_labels=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "neutral        45076\n",
       "positive       21391\n",
       "negative       14021\n",
       "No Majority    10071\n",
       "mixed           3900\n",
       "Total          94459\n",
       "Name: gold_label, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "80488"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Ternary only:\n",
    "\n",
    "train_cat.loc[['positive', 'negative', 'neutral']].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "negative    1200\n",
       "positive    1200\n",
       "neutral     1200\n",
       "Total       3600\n",
       "Name: gold_label, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_distribution(dev, dist_labels=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "negative    1200\n",
       "positive    1200\n",
       "neutral     1200\n",
       "Total       3600\n",
       "Name: gold_label, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_distribution(test, dist_labels=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relationship between validation labels and model predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gold_label  model_0_label\n",
       "mixed       negative           955\n",
       "            neutral           2099\n",
       "            positive           846\n",
       "negative    negative          8677\n",
       "            neutral           3389\n",
       "            positive          1955\n",
       "neutral     negative         11793\n",
       "            neutral          20458\n",
       "            positive         12825\n",
       "positive    negative          3482\n",
       "            neutral           4375\n",
       "            positive         13534\n",
       "dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_model_relationship(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gold_label  model_0_label\n",
       "negative    negative         400\n",
       "            neutral          400\n",
       "            positive         400\n",
       "neutral     negative         400\n",
       "            neutral          400\n",
       "            positive         400\n",
       "positive    negative         400\n",
       "            neutral          400\n",
       "            positive         400\n",
       "dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_model_relationship(dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gold_label  model_0_label\n",
       "negative    negative         400\n",
       "            neutral          400\n",
       "            positive         400\n",
       "neutral     negative         400\n",
       "            neutral          400\n",
       "            positive         400\n",
       "positive    negative         400\n",
       "            neutral          400\n",
       "            positive         400\n",
       "dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_model_relationship(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adversarial rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42,619 of 91,588 examples (46.53%) are adversarial\n"
     ]
    }
   ],
   "source": [
    "adversarial, total = utils.get_adversarial_rate(\n",
    "    train, dev, test,\n",
    "    model_colname='model_0_label',\n",
    "    labels=('positive', 'negative', 'neutral'))\n",
    "\n",
    "print(f\"{adversarial:,} of {total:,} examples ({adversarial/total:0.2%}) are adversarial\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relationship between validation labels and review-level ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gold_label  review_rating\n",
       "mixed       1                 1251\n",
       "            5                 2649\n",
       "negative    1                 3328\n",
       "            5                10693\n",
       "neutral     1                17960\n",
       "            5                27116\n",
       "positive    1                10968\n",
       "            5                10423\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_rating_relationship(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gold_label  review_rating\n",
       "negative    1                600\n",
       "            5                600\n",
       "neutral     1                600\n",
       "            5                600\n",
       "positive    1                600\n",
       "            5                600\n",
       "dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_rating_relationship(dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gold_label  review_rating\n",
       "negative    1                600\n",
       "            5                600\n",
       "neutral     1                600\n",
       "            5                600\n",
       "positive    1                600\n",
       "            5                600\n",
       "dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_rating_relationship(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distribution of validation response distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    36795\n",
       "4    30112\n",
       "5    17481\n",
       "dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_dist_of_majority_dists(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    2356\n",
       "5    1244\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_dist_of_majority_dists(dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    2365\n",
       "5    1235\n",
       "dtype: int64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_dist_of_majority_dists(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Worker distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1833,)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worker_dist = utils.get_global_worker_dist(train, dev, test)\n",
    "\n",
    "worker_dist.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "w135    7528\n",
       "w54     7083\n",
       "w114    6203\n",
       "w22     5896\n",
       "w4      5119\n",
       "dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worker_dist.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45.0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worker_dist.median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    11\n",
       "dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worker_dist.mode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAFLCAYAAAAEZYwbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deVgVZf8/8PdhLwQlJCwQFQxZBI6AmDu4sOmjZoW44op9y600fUpNJTVNy63H59FMfTTFXCu/CAmupGmikhtZCLiWoAgIsnP//vDHfDsewIOBN8L7dV1cXeeee2Y+MweZdzP3zKiEEAJEREREkujJLoCIiIgaNoYRIiIikophhIiIiKRiGCEiIiKpGEaIiIhIKoYRIiIikophhIiIiKSSGkaMjIygVqu1fpYsWaLRLzc3F++++y7atGkDd3d3eHp6IjIyssJlpqenY9SoUXB0dISrqys6deqE2NjYp7E5RERE9AQMZK785ZdfRmJiYpV9ysrK0LdvX+Tm5uLEiROwsLBAXFwcAgMDkZ+fj9GjRyt98/Ly4Ovri1atWuHcuXMwMTHBxo0bERQUhOjoaPTu3bu2N4mIiIiqSSXzCawtW7ZEWlpalX22bt2KoUOHIi4uDj179lTahwwZgpiYGNy4cQPPP/88AGDhwoWYOXMmkpOT4eDgoPTt1KkTMjMz8euvv9bKdhAREdGTq/NjRiIjI2FsbIxu3bpptAcEBODevXuIiYnR6Gtvb68RRMr7Xr58GWfOnHkqNRMREZHupF6mefDgASZMmICffvoJubm5eOmllzBkyBCMGzcOKpUKAHD27FnY2dnB0NBQY97WrVsr0wcOHIjCwkJcunRJ4+xJRX09PT2rrKl8vURERFSzKrsYIzWMGBsbo2fPnvjss8+gr6+P3bt3IywsDEeOHMGWLVsAABkZGXBzc9Oa19zcXJkOAJmZmSgrK1Paq+r7qLVr12Lt2rXK5/T09L+3YY8oKSmBgYHUXd2gcf/Lx+9ALu5/ubj/H3rxxRcrnSZ171y/fl3jc0hICE6ePInPP/8c48ePVy7N6HK2ojxtPcmZjfDwcISHhyvzW1lZVXsZVcnIyKjxZZLuuP/l43cgF/e/XNz/j1fnxox07twZABAfHw8AaNq0KbKzs7X65eTkAIDyBVtaWkJPT0+nvkRERFR3SAsjubm5yM3N1WrX03tYUllZGQDA09MT165dQ3FxsUa/5ORkAIBarQbw8JKPi4sLrly5orXMR/sSERFR3SEtjCxduhTvvfeeVntCQgIAoH379gCA0NBQFBYW4scff9Tot3//flhYWCAwMFBpCw0NRUpKClJSUrT6Ojo6wsvLq6Y3g4iIiP4mqZdptm7diuPHjyufjx49ilWrVqFXr14ICAgAAAwePBjdu3fH9OnTkZWVBQA4ePAgtm/fjqVLl8LU1FSZf8qUKXBycsLkyZNRUFAAANi0aRNOnjyJVatWPcUtIyIiIl1JG8AaFhaGkpISTJw4ESUlJcjNzYWxsTGmTp2KGTNmKANR9fT0sHfvXsyePRs+Pj4wMTGBvr4+Nm3ahMGDB2ss09TUFIcPH8aMGTPg5uYGY2NjmJmZ8emrREREdZjUJ7DWRSqVqtL7oJ8UR1LLxf0vH78Dubj/5eL+f6iq42udu5uGiIiIGhaGESIiIpKKYYSIiIikYhghIiIiqfiwfHpmXe/jrXvfWqxDF82jEiRXQERUd/HMCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEJERERS1ZkwEhMTA5VKhZEjR8ouhYiIiJ6iOhFGCgsLMXHixEqn5+bm4t1330WbNm3g7u4OT09PREZGVtg3PT0do0aNgqOjI1xdXdGpUyfExsbWVulERET0NxnILgAAPv30U7i5uSE5OVlrWllZGfr27Yvc3FycOHECFhYWiIuLQ2BgIPLz8zF69Gilb15eHnx9fdGqVSucO3cOJiYm2LhxI4KCghAdHY3evXs/zc0iIiIiHUg/M5KWloYVK1Zg2bJlFU7ftm0bjhw5gsWLF8PCwgIA0KtXL4SEhGDatGl48OCB0nfFihVISkrCypUrYWJiAgAYOXIkfHx8qjzzQkRERPJIDyNTpkzB5MmT0aJFiwqnR0ZGwtjYGN26ddNoDwgIwL179xATE6PR197eHg4ODlp9L1++jDNnztT8BhAREdHfIjWMREdH48KFC5g+fXqlfc6ePQs7OzsYGhpqtLdu3VqZDjwcd3Lp0iWtIFJRXyIiIqo7pIWRwsJCTJo0CcuXL4exsXGl/TIyMmBubq7VXt6WkZEBAMjMzERZWZlOfYmIiKjukDaAdfHixXByckLfvn0f21elUj22jxBC576PWrt2LdauXat8runQUlJSwiDUwDX075//BuTi/peL+//xpISRtLQ0LF++HAkJCY/t27RpU2RnZ2u15+TkAACsrKwAAJaWltDT09Op76PCw8MRHh4O4GGYqazfk8rIyKjxZRJwXXYB1dDQv3/+G5CL+18u7v/HkxJGYmNj0ahRIwwcOFBr2vfffw+1Wg0DAwMkJCTA09MTsbGxKC4u1hg3Un4bsFqtBgAYGxvDxcUFV65c0Vrmo32JiIio7pAyZmTcuHG4du0aEhMTNX4AoF+/fkhMTFTOmoSGhqKwsBA//vijxjL2798PCwsLBAYGKm2hoaFISUlBSkqKVl9HR0d4eXnV8pYRERFRdUm/tfdxBg8ejO7du2P69OnIysoCABw8eBDbt2/H0qVLYWpqqvSdMmUKnJycMHnyZBQUFAAANm3ahJMnT2LVqlVS6iciIqKq1YknsP718kn5ZZp+/fohIiICenp62Lt3L2bPng0fHx+YmJhAX18fmzZtwuDBgzWWY2pqisOHD2PGjBlwc3ODsbExzMzM+PRVIiKiOkwlym9DIQAPB7DW9C7h4KXacb2Pt+wSdNY86vGDtesz/huQi/tfLu7/h6o6vtb5yzRERERUvzGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQGunYsKirCnTt3YGZmBjMzMwDA5s2b8csvv8Df3x/+/v61ViQRPT3X+3jXznJrYZnNoxJqYalE9LTpfGZk8eLFcHBwwJo1awAAK1euRFhYGL788kv06dMHO3bsqLUiiYiIqP7SOYzs27cPcXFxmDZtGgBg6dKlCAgIQGZmJg4fPozPPvusWiu+du0aZs6cCR8fH3h6esLV1RWurq5YsGABCgoKNPqmp6dj1KhRcHR0hKurKzp16oTY2NgKl3vlyhUMHDgQjo6OcHZ2Ru/evXH69Olq1UZERERPj85hpLi4GJ07dwYA/Pzzz7hx4wamT58OfX19dO7cGYWFhdVa8cGDB7F8+XIsXboUZ86cwcWLF7FmzRrMnz8fY8eOVfrl5eXB19cX6enpOHfuHC5evIjw8HAEBQVpBZJbt26hS5cuaNq0KZKSkpCUlIQuXbqge/fuOH/+fLXqIyIioqejWmGk3DfffIOXXnoJfn5+SptKparWips1a4b3338f3bp1U9q6dOkCf39/7NmzR2lbsWIFkpKSsHLlSpiYmAAARo4cCR8fH0ycOFFjmXPnzkVOTg4+++wz6OvrAwBmzZoFCwsL5YwOERER1S06hxFra2t88skn2LFjB9atW4chQ4Yo05KSkiCEqNaKAwMDMXfuXK32nJwc2NraKp8jIyNhb28PBwcHjX4BAQG4fPkyzpw5AwAoKyvD9u3b8eqrryoDbAFAX18fvXr1QlxcHDIyMqpVIxEREdU+ncPIggULsGTJEgwaNAhmZmbKmYbFixejW7duCAgI+FuF5OfnY/ny5Thz5gxWrlwJACgsLMSlS5e0gggAtG7dGgBw9uxZAEBKSgqys7Mr7VtWVoZz5879rRqJiIio5ul8a2/79u2RlpaGpKQkuLq6olGjRgCAoKAgeHt7w93d/YmL8PLywvnz52FjY4Ovv/5aCTaZmZkoKyuDubm51jzlbeVnO8r/q0tfIiIiqjt0DiMhISEAHt7SWx5EAPytEFLu9OnTKCkpwXfffYehQ4di2LBhWL16tXLpR5fxKNXp+6i1a9di7dq1yueaDi0lJSUMQg0cv//awf2qG/4Nkov7//F0DiO7d+/GqlWrYGFhUTuFGBjg9ddfR1JSEmbPno3+/fvD19cXenp6yM7O1uqfk5MDALCystL4ry59HxUeHo7w8HAAD8NMZf2eVEZGRo0vk2rnIVq15Vn6/rlf6x/+DZKL+//xdB4z4uHhgf/5n/+BsbFxjaw4Pz8fpaWlWu3t2rUDAJw6dQrGxsZwcXHBlStXtPolJycDANRqNQDAwcEB5ubmlfbV09OrkbM4REREVLN0DiP29vZIT0+vdHpQUFC1VhwUFIRvvvlGqz0tLQ0A0LRpUwBAaGgoUlJSkJKSotFv//79cHR0hJeXFwBAT08PISEhOHHiBHJzc5V+ZWVlOHDgAHr16sVkSkREVAfpHEbee+89jBgxArt27cLly5dx69YtjZ8bN25Ue+ULFy7UCBlnz57FggUL0Lx5cwwaNAgAMGXKFDg5OWHy5MnKk1k3bdqEkydPYtWqVRrLmzdvnnKnT/lZlwULFiAzMxNLliypdn1ERERU+3QeM1L+9NXKHsNeXZ988gk2btyIfv36QU9PDw8ePAAAvPbaa8qDygDA1NQUhw8fxowZM+Dm5gZjY2OYmZkhOjoavXv31ljmyy+/jGPHjuH999+Hs7Mz9PT0YGtriyNHjvASDRERUR2lcxhp3rw5PvroowqnCSEwf/78aq24Y8eO6Nixo059ra2tsXHjRp36Ojg4YPfu3dWqhYiIiOTROYx4eXlhzJgxlU4/fvx4jRREREREDYvOY0Yed7Zh/fr1f7sYIiIianh0DiPlEhISsGjRIsyaNQsAcOLEiWq/sZeIiIioXLXe2hsSEoIOHTrgww8/VM6EbNiwAWq1Grdv3661IomIiKj+0jmMREREICEhAatWrcLRo0eV54CsWbMGYWFhlQ5uJSIiIqqKzgNYt2/fjsOHD8POzu7hjAb/N+v06dPh7e1d89URERFRvafzmREDAwMliGgtRE8PRUVFNVYUERERNRw6h5GCggLcunVL+Vz+llwAuHnzJvLz82u2MiIiImoQdA4jAwYMgK+vL7Zs2YLr169DpVIhKysLcXFx6NevH0JCQmqzTiIiIqqndB4zEhERgZ9++gnDhw+HSqUCAFhaWgIAOnXqxAGsRERE9ER0DiOmpqY4evQoNm3ahB9++AEZGRmwsrJCUFAQhg0bpjGglYiIiEhX1UoQBgYGGD16NEaPHq01bceOHXjzzTdrrDAiIiJqGHQeM9KzZ88qp1f3RXlEREREQDXCSFpaGkpLS7Xas7KyMGjQIFy4cKFGCyMiIqKGQecwcuPGDQwaNAhlZWVK2w8//IC2bdvi+++/R6NGjWqlQCIiIqrfdB4z4uLiAldXVwwfPhxr1qzB9OnTsWbNGri5uSE6OhrvvPNObdZJ9Ey73odPKCYiqozOYSQ+Ph6NGjXCtGnTYGNjgwcPHmDq1KmYP38+jIyMcPTo0dqsk4iIiOopncNI+WWYpUuXIj8/H3/88Qc+/fRTZfqAAQPw7bff1nyFREREVK9VGkYWLlxY6Uwvv/wydu7ciREjRsDJyQkAcOrUqZqvjoiIiOo9lfjrS2b+Qk9P57GtDxekUlV4t82zRqVSoZJd8sTKHxBHNYvjMOhZ0jwqQdq6+TdILu7/h6o6vlaaODw8PFBcXKzTT1FREdzd3WttA4iIiKj+qvQyzYQJE6Cvr6/zgubMmVMjBREREVHDUumZkTFjxmh8trKywosvvoiUlJQK+w8YMKBmKyMiIqIGQee7aQoKCnDy5EnY29vXZj1ERETUwOg8SrVt27ZwcXGpdPrNmzdrpCAiIiJqWHQOI6+++ip+/vnnSqf/4x//qJGCiIiIqGHR+TKNq6srRowYgd69e6Nt27YwMzPTmH7v3r0aL46IiIjqP53DSHh4OADgt99+q3C6SqWqmYqIiIioQdE5jDg7O+P777+vcJoQAv3796+xooiIiKjh0DmMDB06FA4ODpVO51t7iYiI6EnoPID1ww8/rHL622+//beLISIiooanWi+guX//PubNm4eOHTvCwcEBHTt2REREBO7fv19b9REREVE9p/Nlmjt37qBLly747bffYGBgAAsLC1y/fh0nT57Etm3bEB8fD0tLy9qslYiIiOohnc+MzJw5E9bW1jh58iSKiopw+/ZtFBUV4eTJk7C2tsbMmTNrs04iIiKqp3Q+MxIbG4tz586hUaNGGu3t27fHd999Bw8PjxovjoiIiOo/nc+MmJiYaAWRcubm5jAxMamxooiIiKjh0DmMlJWV4fLlyxVO+/XXX1FWVlZjRREREVHDofNlmmHDhqFHjx6YOnUqXn31VbzwwgvIzMzEiRMn8Pnnn2PChAm1WScRERHVUzqHkQ8++ACJiYmYNm2axqPfhRAICQnB9OnTa6VAIiIiqt+qDCM7duzAm2++CQDQ19fHzp07ceDAAfzwww+4c+cOrKysEBQUBF9f36dRKxEREdVDKiGEqGyii4sLEhMTYWRkVKMrTUlJwVdffYXY2FiUlJQgPz8f1tbWePfdd7XecZObm4vZs2dj3759MDY2hoGBAd5//30MHjxYa7np6emYMWMGjh07BkNDQzRu3Bjz5s1D7969da5NpVKhil3yRDIyMmBlZVWjyyTgeh9v2SUQ6ax5VIK0dfNvkFzc/w9VdXyt8szIjRs34OTkBBcXF/Tp0wfBwcFo0aLF3y7o7bffxvXr1xEdHQ07OzsIIRAREYEBAwZg6dKlmDp1KoCHg2b79u2L3NxcnDhxAhYWFoiLi0NgYCDy8/MxevRoZZl5eXnw9fVFq1atcO7cOZiYmGDjxo0ICgpCdHR0tQIJEVFNkx2er1ejr8zgRA1TlWdG2rVrh7NnzyIhIQHR0dGIjo5Gbm4uAgMDERwcjK5du0JfX7/aKw0MDMSoUaMwaNAgpU0IgebNm6OoqAjp6ekAgK1bt2Lo0KGIi4tDz549lb5DhgxBTEwMbty4geeffx4AsHDhQsycORPJyckaL/Tr1KkTMjMz8euvv+pUG8+MPDtk/3Enqq8YRmoWjwEPVXV8rfLW3r179wIAvL29MXv2bBw/fhyHDh2CWq3GunXr0KZNG7z55pvYsGED/vzzT50L2rt3L0JCQrSKtLGxQXZ2tlJsZGQkjI2N0a1bN42+AQEBuHfvHmJiYpS2yMhI2Nvba71ZOCAgAJcvX8aZM2d0ro+IiIienirDiK2trVabpaUlhgwZgnXr1mHZsmVITU3F2LFjK+xbGUNDQ407cgCgpKQEycnJ6NGjhzLt7NmzsLOzg6GhoUbf1q1bK9MBoLCwEJcuXdIKIhX1JSIiorpF51t7AeDatWuIiopCVFQUDh06hIKCAggh4OTkhODg4L9VyJYtW5CXl4dFixYpbRkZGXBzc9Pqa25urkwHgMzMTJSVlSntVfUlIiKiuqXKMFJWVoZjx44pAeTSpUsQQuC5556Dn58fgoODERwcjJYtW/6tIn7//XdMnToV69at03rHzaNnUCpSfllHl74VWbt2LdauXat8rungUlJSwjBERM8M/r2qWTwGPF6VYcTS0hI5OTkAAHt7e7zzzjsIDg6Gn58fjI2Na6SA1NRU+Pv7Y/HixRg2bJjGtKZNmyI7O1trnvKaygcEWVpaQk9PT6e+FQkPD0d4eDiAh4GmpgcacfBS7ajO3QFEpDv+vapZPAY8XpVhxM/PDwcOHIC9vT0mTZqEoKAgNGvWrMZWnpSUhODgYCxatEjjzppynp6eiI2NRXFxsca4keTkZACAWq0GABgbG8PFxQVXrlzRWsajfYmIiKhuqXIA6+7du3H37l18/vnnuHjxInr27Alvb2989NFHOHHixN+6BfbMmTMIDAzE6tWrNYLI+PHjcfr0aQBAaGgoCgsL8eOPP2rMu3//flhYWCAwMFBpCw0NRUpKClJSUrT6Ojo6wsvL64lrJSIiotpT5XNGKpKWlqaMIblw4QK6d++O4OBgBAYGwsLCQqdlxMfHo2/fvggNDUXXrl01pi1YsAD//ve/4evri7KyMvTo0QN5eXmIjY1FkyZNcPDgQQQEBGDNmjVaDz3z9vZG69atsWPHDpiYmGDTpk0YPXo09u3bB39/f51q43NGnh18zghR7eBzRmoWjwEPVXV8rXYYKVdYWIioqCh8+OGH+P3336Gvr4+ioiKd5vX09KzyVttDhw4p77u5f/++8jh4ExMT6OvrY/r06RU+Dv727dvK4+CNjY1hZmaGiIgIPg6+nmIYIaodDCM1i8eAh544jMyfPx+zZs1SPl+9ehX79u3Dvn37cOjQIeTn50MIgZYtWyI4OBhffPFFzVf/lDGMPDsYRohqB8NIzeIx4KEnfjfN9u3b0aVLF+zbtw9RUVH49ddfIYSAoaEhOnfujODgYPTp0wfOzs61UjgRERHVf1WGkQsXLqBnz54QQuCll17CyJEjERwcDH9/f5iZmT2tGomIiKgeqzKMNGnSBO+99x6Cg4Ph6en5tGoiIiKiBqTKMKJWqzXGjBARERHVtCqfM3Lw4MGnVQcRERE1UFWGESIiIqLaxjBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJZSC7ACIiqluu9/GWXYLOmkclyC6BagDPjBAREZFUDCNEREQkFcMIERERScUwQkRERFJJDSMPHjzA5MmToaenh8OHD8sshYiIiCSRFkaOHTuGdu3a4ejRoxBCVNovNzcX7777Ltq0aQN3d3d4enoiMjKywr7p6ekYNWoUHB0d4erqik6dOiE2Nra2NoGIiIhqgLRbe+fMmYMvv/wSKSkpGDVqVIV9ysrK0LdvX+Tm5uLEiROwsLBAXFwcAgMDkZ+fj9GjRyt98/Ly4Ovri1atWuHcuXMwMTHBxo0bERQUhOjoaPTu3ftpbRoRERFVg7QzIzExMejWrVuVfbZt24YjR45g8eLFsLCwAAD06tULISEhmDZtGh48eKD0XbFiBZKSkrBy5UqYmJgAAEaOHAkfHx9MnDix9jaEiIiI/hZpYcTA4PEnZSIjI2FsbKwVWgICAnDv3j3ExMRo9LW3t4eDg4NW38uXL+PMmTM1UzgRERHVqDp9N83Zs2dhZ2cHQ0NDjfbWrVsr0wGgsLAQly5d0goiFfUlIiKiuqVOh5GMjAyYm5trtZe3ZWRkAAAyMzNRVlamU18iIiKqW+r8u2lUKtVj+5TfjaNL34qsXbsWa9euVT7XdHApKSlhGCIiqgXPwt9WHgMer06HkaZNmyI7O1urPScnBwBgZWUFALC0tISenp5OfSsSHh6O8PBwAA8DTVV9n0RGRkaNL5OA67ILICLpnoW/rTwGPF6dvkzj6emJa9euobi4WKM9OTkZAKBWqwEAxsbGcHFxwZUrV7SW8WhfIiIiqlvqdBgJDQ1FYWEhfvzxR432/fv3w8LCAoGBgRp9U1JSkJKSotXX0dERXl5eT6VmIiIiqp46HUYGDx6M7t27Y/r06cjKygIAHDx4ENu3b8fSpUthamqq9J0yZQqcnJwwefJkFBQUAAA2bdqEkydPYtWqVVLqJyIioseTNmZk5cqVWL9+PTIzMwEAY8eORaNGjTBr1iy88cYbAAA9PT3s3bsXs2fPho+PD0xMTKCvr49NmzZh8ODBGsszNTXF4cOHMWPGDLi5ucHY2BhmZmZ8+ioREVEdpxJVvRimAVKpVFW+K+dJcPBS7bjex1t2CUQkWfOoBNklPBaPAQ9VdXyt05dpiIiIqP6r07f2EhERVeVZOUN6Hc/GWRxZeGaEiIiIpGIYISIiIqkYRoiIiEgqhhEiIiKSimGEiIiIpGIYISIiIqkYRoiIiEgqhhEiIiKSimGEiIiIpGIYISIiIqkYRoiIiEgqhhEiIiKSimGEiIiIpGIYISIiIqkYRoiIiEgqA9kFUN1yvY+37BKIiKiB4ZkRIiIikophhIiIiKRiGCEiIiKpGEaIiIhIKoYRIiIikophhIiIiKRiGCEiIiKpGEaIiIhIKoYRIiIikophhIiIiKTi4+CfgoKRQbguuwgiIqI6imdGiIiISCqGESIiIpKKl2mIiIiegmfprejNoxKe6vp4ZoSIiIikYhghIiIiqRhGiIiISCqGESIiIpKKYYSIiIikYhghIiIiqRhGiIiISCqGESIiIpKKYYSIiIikqndhpLi4GBEREXB2doa7uzvatm2LVatWQQghuzQiIiKqQL17HPyYMWNw9OhRxMfHo3nz5jh37pHc1WgAABgrSURBVBy6dOmCjIwMREREyC6PiIiIHlGvzowcP34cmzdvxuzZs9G8eXMAgLu7OyZMmIBFixbh2rVrkiskIiKiR9WrMBIZGQkA8Pf312gPCAhAcXExdu7cKaMsIiIiqkK9CiNnz56FkZERbG1tNdpbt26tTCciIqK6pV6NGcnIyICZmRlUKpVGu7m5uTK9ImvXrsXatWuVz4/OT0RE1KA85eNgvQojQognChLh4eEIDw+vhYoe8vb2RkJCQq0tn6rG/S8fvwO5uP/l4v5/vHp1mcbKygo5OTlat/Hm5OQo04mIiKhuqVdhxNPTE0VFRbhx44ZGe3JyMgBArVbLKIuIiIiqUK/CSGhoKAAgLi5Oo33//v0wNDTEG2+8IaOsWr0ERI/H/S8fvwO5uP/l4v5/PJWoZ48mHTFiBOLj4xEfHw9bW1ucP38eXbp0weTJk/nQMyIiojqo3oWR4uJiLFy4EJGRkTAyMkJpaSnGjx+PiRMn8i4ZIiKiOqjehREiqntiYmIQFBSEsLAwbNy4UXY5RFTH1KsxI3UJX9gnT0pKCmbOnAkfHx94enrC2dkZvr6++O6772SX1iAVFhZi4sSJsstokI4ePYrg4GDl30GrVq0wZMgQ5Obmyi6t3ktJSUFYWBicnZ3h4eEBV1dXTJkyBXfu3JFdWt0kqFYMHz5ctGjRQly7dk0IIcQvv/wizMzMxOzZsyVXVv8FBAQIFxcXcfXqVSGEEGVlZWLu3LkCgFi6dKnk6hqeiIgI8dprrwkAIiwsTHY5DcbmzZuFjY2NSEhIUNr27t0rAIg//vhDYmX1X1ZWlmjWrJno2rWryMrKEkIIcevWLdGmTRuhVqtFaWmp5ArrHoaRWnDs2DEBQKxbt06j/YMPPhCGhobKQZJqR0BAgNi2bZtGW1lZmbCxsRFWVlaSqmqYUlNThaWlpUhLS2MYeYpu3bolTE1NRWRkpNa0vXv3ioKCAglVNRw7d+4UAMSuXbs02j/77DMBQCQmJkqqrO7iZZpawBf2ybV3716EhIRotKlUKtjY2CA7O5uXyp6iKVOmYPLkyWjRooXsUhqUTZs2IT8/H/369dOa1rdvXxgbG0uoquEofwVJcXGxRnv5Z+5/bQwjtYAv7JPL0NBQ686pkpISJCcno0ePHryr6imJjo7GhQsXMH36dNmlNDjx8fFo1qwZTp8+jaCgILi6usLV1RWTJk1Cenq67PLqvR49eqBXr15YvHgxrl+/DgBISkrCv/71L/Tt2xdOTk6SK6x7GEZqwZO+sI9qz5YtW5CXl4dFixbJLqVBKCwsxKRJk7B8+XL+X6AEV69eRWZmJsaOHYslS5bg4sWL2LNnD/bv34+OHTvi3r17skus1/T19bFnzx54eHigVatWsLGxQbt27TB8+HB8++23ssurkxhGaoF4whf2Ue34/fffMXXqVKxbtw4eHh6yy2kQFi9eDCcnJ/Tt21d2KQ1Sfn4+CgoKMHPmTLRt2xYA4OjoiE8//RQpKSlYvXq15Arrt6tXr8LT0xPp6en4448/cPPmTZw5cwa7du1CWFgYSktLZZdY59Srt/bWFVZWVrh69apWKOEL+56+1NRU+Pv7Y/HixRg2bJjschqEtLQ0LF++nG8plaj8LGy7du002tu3bw8AOHHixFOvqSH54IMP8Pvvv+PIkSPK33sXFxd88sknGDhwILp3745x48ZJrrJu4ZmRWsAX9tUNSUlJ6NGjBxYtWoQxY8bILqfBiI2NRaNGjTBw4ECo1WrlBwC+//57qNVqeHt7S66yfnNxcQEAlJWVabTr6+tX2E416+zZszA3N8dLL72k0V4+VoRhUBvDSC2oqy/sa0jOnDmDwMBArF69GoMGDVLax48fj9OnT0usrP4bN24crl27hsTERI0fAOjXrx8SExN51qSWDRw4EACU/V6u/POrr7761GtqSJo1a4acnBxkZmZqtKempgLg2fGKMIzUgs6dO2P48OGIiIhQzo6cP38eX3zxBf75z3/yNsdaFh8fDz8/PwQGBuLu3bv4+uuvlZ+jR4/i/v37skskqlWvvfYaevbsiQULFih3c9y5cwcffvghWrVqhQkTJkiusH6bOnUqVCoVJk2ahIKCAgDAn3/+idmzZ8PMzAxjx46VXGHdw3fT1BK+sE8eT0/PKm+fPnToEHx9fZ9eQQ1c+SWaX375BRYWFrCzs0O/fv34Fu1alpeXh4iICOzcuRNGRkbIz8+Hv78/Pv74Y1hbW8sur9778ccfsWjRIiQnJ8PIyAhFRUXo0KEDZs6cCUdHR9nl1TkMI0RERCQVL9MQERGRVAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEL0jDp//jzUajUaNWr0VB/itmHDBqjVaqhUKsydO7fa87/22muws7ODSqVCWlpajdRUG8t855130Lp1a6hUKhw+fFjn+QoLCzFr1izk5uZK+45kWrBgAVxcXKBSqbBx48Zqz5+YmIgvv/yy5gujOo1hhOqVrKwsqNVqNGvWDCqVCi4uLhova1Or1XBxcUHLli1ll/q3ubm5ITEx8am/dG7UqFFa7zypjj179tT401drY5n/+te/sG7dumrNk5ubq4SORo0aSfuOZJo5cyb27dv3xPO7uLjgf//3fxEeHl6DVVFdxzBC9UqTJk2QmJiIt956CwCwb98+rRe2/Z0/lERVGT9+PKysrDB//nzZpTyzjIyM8M033yAqKgpr1qyRXQ49JQwj1OBYW1tj/fr1ssugeiYxMRFbt27FnDlzZJfyzDMxMcHUqVMxZ84cFBUVyS6HngKGEWpQfH19cfLkSfTo0QMA8M033yjjAqytrbFy5UqUlpZCrVbD0NAQzs7O+P777wEAly5dQlhYGNzc3NCuXTt4eHjgvffeQ05OjrL8S5cuaYwR2L59O3x9fWFraws/Pz8kJyfj7t27GDZsGNzc3PDKK69g8+bNyvx3796FWq3GCy+8gJYtW2LPnj3o1KkTHB0dYWtri48//hi6vk5q165daN++PVq3bo2WLVuif//+uHTpkkaf5ORkvP7662jbti08PT3h5eWFGTNmICMj44n27/Lly9GxY0d4eXlBrVaja9euOHDgQKX909LSMGDAALi7u8PS0hJhYWG4d++eRp+SkhIsWLAAbdq0gZOTE+zt7fH2229r9atMamoqQkNDYWdnhzZt2kCtVlc4luHy5csICgrCCy+8AC8vL4wYMULrFfBV+frrr2FlZQUvLy+d59m9ezc6dOgABwcH2NnZoU+fPhW+5HHfvn1o164drK2t4e3tjalTp+KDDz6ASqWCWq3Ghg0bqlzPF198AXd3d+X3tl+/fvj22281+ly9ehVDhw5FixYt4OHhATc3N4wbNw4JCQlKn6ioKPTp0wceHh5Qq9Xw9PTE6tWrdf6dzMvLw7Rp02Bvbw9nZ2e88sor+PDDD1FYWKjVNyAgALdv30ZsbKxOy6ZnnCCqh+bMmSMAiNTUVI327t27i0OHDmm0FRQUiLZt2wobGxuRkZEhhBBix44dolevXqKsrEzpt2rVKuHn5ydyc3OFEELk5uaK/v37i4EDB2qtv3v37sLa2losWrRIWUeHDh2Et7e3eP/990VWVpYQQohly5YJfX19kZKSojF/WFiYMDY2FiEhIaKgoEAIIcS+ffuEgYGBmDFjhta6unfvrtH273//WwAQ//3vf4UQQpSWlorw8HDRpEkTjX3i6OgoRo0aJUpLS4UQQly+fFk0bdpUax9VBICYM2eORlvTpk3Fvn37lM8HDhwQzz//vDhz5oxGvw0bNggAolu3buLGjRtCCCGuXLkibG1tRadOnZR6hBBi0KBBwsLCQiQmJgohhLh7965o37698Pb2FkVFRVrL/Ov2Xb16VVhZWYng4GDx4MEDIYQQBw8eFIaGhmLZsmVKvz///FO8+OKLomvXriI7O1sIIcS5c+dE27ZtBQCd9oePj4/o1q1bhdMq+o5Wr14tVCqV2LZtmxBCiLKyMjFt2jTx3HPPiZMnTyr94uLihL6+vnj33XeV38fIyEhhamoqdPkTvmXLFmFoaCguX74shBCipKRETJkyRaOea9euCWtra/H6668rv2/Xr18Xr7zyiggLC1P6vf766+Kf//yn8v1cvXpV2Nvbi5UrV2qsMzU1VQAQGzZsUNqKi4tFly5dRKtWrURaWprG/AMGDNCqu7i4WOjp6Wn9vlP9xDBC9VJ5GHF2dhYeHh7Kj6mpaYUHlsTERGFkZCT69OkjUlNTRatWrcT169c1+qSnp4vbt29rtEVHRwsAIj09XaO9e/fuwsLCQuNguWTJEgFA7NixQ2OZAMS6des05g8LCxMAxNWrVzXa+/fvL4yMjDTW9+iB7v79+8Lc3Fx07dpVY96srCxhYmIixo0bJ4QQIiMjQwAQ8+fP1+i3fv168fvvv2vto0dVFEbOnz+v1a9Dhw7i7bff1mgrDw7lYancsmXLBADx7bffCiGEOHLkiAAgZs+erdEvJiZGABBbtmzRWuZfw8ioUaMEAHHlyhWN+UNDQ0Xjxo2VgDJjxgwBQJw4cUKjX/nvkS5hxNLSUrzxxhsVTnv0O8rJyRFmZmbC399fo19hYaFo2rSp6NSpk9LWoUMHYWFhIfLz87WWqUsYeeedd4S5ubmyrUIIcevWLbFq1Srl86hRo4S+vr4SDMt99dVXYvz48crnK1euaCxHiIf7zsXFRaOtojDy3//+VwAQX331lUbf//znPwKAOHbsmFbtVe1Tql94mYbqtUcHsFZ2V4OHhwciIiIQFRWFDh06YOHChbC1tdXoY25ujq1bt6JLly5wdXWFWq3GhAkTADy83PEoBwcHGBoaKp8tLS0BAM7Ozkpb06ZNAQB//PGH1vxNmjSBnZ2dRturr76KoqIiHDt2rNJtPn78OHJyctC9e3eN9saNG6NFixaIi4tT6vH09MS8efMwduxYHDx4EMXFxRg1ahRat25d6fKrUlJSgiFDhsDDw0M5lX/hwoUK9w/wcL8/un0AcOjQIQBATEwMAGhtS/l85dtSmZiYGNja2sLe3l5r/uzsbJw6dUpZn76+vtbvh1qtrnL5f5WdnY3nnntOp77Hjx/H/fv30bFjR412IyMjeHp64qeffkJ2djby8vJw6tQpeHh4wMTE5Ilq6927N+7fvw9vb2988cUXuHHjBl566SXldxcAoqOjYWtrCxsbG415R48ejf/85z/KZ1NTU8ycORPt27eHm5ubcsmrsu/3r57ku3zuueeQnZ2t03bSs81AdgFET1NVz4t4//33sXnzZqSkpKBr165a08ePH4/IyEhERUWhV69eyvL8/PwqvOZtamqq8VmlUmm1l7eVlJRozd+4cWOtthdeeAHAw7EllSkf77Fx40bs3btXY9qDBw+UdapUKhw6dAjLli3Dpk2b8NVXX8HS0hJvvfUWPvroIxgZGVW6jopcvHgRnTt3hr+/P+Lj42Fubg7g4TidivZPRdv46PaVb8uECRNgbGys0dfa2hr5+flV1pSRkQEjIyOtA3d+fj6sra2VMSF3795F48aNoa+vr9GvSZMmVS7/r/T09HQeO3Hnzh0A/7e9f2VpaQkhBO7cuQNjY2OUlZVV2E/X2vr3748DBw5gxYoVmDp1KiZOnIiuXbti6dKl8PHxUepxc3OrcjkPHjyAn58fiouLERMTAwcHBwDA3LlzMW/evMfWUf5dDhw4UPkdBIDS0lJYW1sjNzdXax4hBPT0+P/MDQHDCNH/9/PPP8PAwABGRkYYOXIk9u/fr/zRLCgowJYtWxAQEKAEkdpW0f8Rlh+ky8+yVMTKygrAwwP4jBkzqlyHubk55syZgzlz5uDnn3/GypUrsWDBAqhUKnz88cfVqnfr1q148OABZs6cqQSRx3l0Gx/dvvJt2bhxIzp06FCtesrnb9y48WOfi2JpaYnU1FSUlpZqBBJdB8kCD4NFXl6eznUBqHCA7N27d6FSqdC0aVMYGBhAT0+vwvBZndr8/Pzg5+eHrKwsbN++HfPmzUPv3r2RlpYGCwsLWFlZPXaw7rFjx5CUlIQlS5YoQaQ6yrd5//79sLa21mmevLy8CoMY1T+MnNQglV8eKZeXl4dx48Zhy5YtWLlyJeLi4rBq1SpleklJCUpLS7X+L62iyys1JSsrC9euXdNoO3HiBIyMjNC5c+dK5+vUqRPMzc1x+vRprWnlByIASE9Px6RJk5RpPj4++Prrr9G2bVv88ssv1a63/BbM6uyjR9dz4sQJAFDudgoKCgKACrflo48+wq5du6qsKSgoCKmpqVoH7jt37mDgwIHKAbhnz54oKytTLttUVl9VnJycdP59KP+Oyre3XFFREc6ePYuOHTuicePGMDU1RYcOHfDLL7+goKDgiWpbvny5sp4mTZogPDwcK1euRE5ODlJTUwE83E83btzAzZs3Neb9+uuvMW7cOKU2oHrf719V9V2+9dZbiI+P12grKChAVlYW2rRpo9Py6RknecwKUa2o7G6aco/+6o8bN06sWLFC+Txw4EBhYmIiLl26pLT16tVLmJiYiFOnTgkhhMjMzBReXl4VDnCs6O6JigZYltfy6EDQsLAw0ahRIzF8+HCtu2n++c9/PnZda9asEfr6+mLnzp1KW2JiorCxsRFRUVFCiIeDDA0NDcXBgweVPr/99pto3LixxuDGyjxad3x8vFCpVGLw4MHKwN3yQYuV7QsvL6/H3k0zZMgQYWtrqzE4dsuWLaJZs2Yag4wr2r/ld4kMHjxYGXiZk5Mj3nzzTTF48GClX3p6umjWrJno2rWryMnJEUI8HIzbsmVLnQewzps3T5ibm2vUXq6y70hPT09s375dCPHwbprp06dr3U1z6NChCu+msba21mkAa1hYmOjfv79yF1hJSYmYOHGisLGxUQbFVnQ3TUpKimjRooXYtWuXEOLh77uVlZVwdnZWBnKfP39emJuba9VR2d003bp1E25ubsrA7NLSUrFkyRLRunVrpb5yP//8swAg4uPjH7uN9OxjGKF6JTMzU7Ro0UI0btxYABA2NjaiRYsWWj/lfzxPnTol3N3dBQDh7u4ubt++LQ4ePCgcHBwEAGFlZSXGjBkjhBDi9u3bYujQocLa2lp4e3uLwMBAMXfuXAFAODg4iHnz5ombN28qd+2YmpoKDw8Pcfv2bfH222+L5s2bK3f4fPnllyImJkZ4eHgIAMLa2loEBQUp2xEWFiZatGgh4uLiRLdu3cQrr7wiXn75ZREREaEckM6dO6e1rosXLyrL2LNnj+jQoYOws7MTXl5ewtfXV0RHRyvTHzx4IObNmye8vLyEh4eHUKvVwsPDQyxfvrzKfbx+/XqNujt27KhMi4yMFK6ursLGxkb4+fmJadOmCW9vb419MWDAAGVf7N27V/Tu3Vu0bdtWWFhYiBEjRojMzEyN9ZWUlIhFixaJNm3aiFdeeUWo1Wrx5ptviqSkJKXPX5fp7OwsPvnkE2VaWlqaGDp0qLC1tRXu7u5CrVaLuXPnisLCQo31/PbbbyIoKEhYWFgIDw8P8dprr4nNmzcr3++jdwQ96urVq8LAwEDs379faXvcd7R7927h4+Mj7O3tRfPmzUVQUJA4ffq01rKjo6OFWq0W1tbWwsfHR3z00Udi1qxZOoWRo0ePipCQEOHi4iLUarVwdXUVb7zxhvj11181+qWmpoohQ4aI5s2bC3d3d9G+fXvxzTffaPQ5ffq08PX1VW6DHjx4sHLnl4eHh4iKihLz588Xzs7OAoBo3ry5eP3115X58/LyxIwZM4S9vb1yp9uYMWPEzZs3ter+8MMPRZs2bTRur6f6SyWEjiOuiOipGTlyJA4fPlxjL32jp2PWrFnYv38/fvrpJ63BsDVt4sSJ2LhxI+7fv1+r65EhPT0djo6O2LZtGwIDA2WXQ08Bx4wQEdWQiIgItG3bFm+88QZKS0trZJkJCQkVDiZOTEys1tNenxWZmZkICAjAggULGEQaEN5NQ0RUQ/T09LB+/Xrs3r0bRUVFOj93pCpZWVlYunQp/vGPfyi3KG/atAnHjx9Xnt1Rn9y7dw9r1qxRbjumhoGXaYjqkLt376Jnz564du0acnNz4eLigmXLlsHPz092aSTJjRs3MHfuXMTHx8PExARZWVmws7PD7Nmz4e/vL7s8ohrBMEJERERSccwIERERScUwQkRERFIxjBAREZFUDCNEREQkFcMIERERScUwQkRERFL9P9hSmyQElqEgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "utils.plot_global_worker_dist(\n",
    "    worker_dist, os.path.join('img', 'round1-workers.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimates of human performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.616"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_fleiss_kappa(dev).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.615"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_fleiss_kappa(test).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>support</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>positive</th>\n",
       "      <td>0.884</td>\n",
       "      <td>0.879</td>\n",
       "      <td>0.881</td>\n",
       "      <td>1200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>negative</th>\n",
       "      <td>0.931</td>\n",
       "      <td>0.857</td>\n",
       "      <td>0.892</td>\n",
       "      <td>1200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>neutral</th>\n",
       "      <td>0.860</td>\n",
       "      <td>0.872</td>\n",
       "      <td>0.866</td>\n",
       "      <td>1200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Macro avg</th>\n",
       "      <td>0.892</td>\n",
       "      <td>0.869</td>\n",
       "      <td>0.880</td>\n",
       "      <td>1200.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           precision  recall     F1  support\n",
       "positive       0.884   0.879  0.881   1200.0\n",
       "negative       0.931   0.857  0.892   1200.0\n",
       "neutral        0.860   0.872  0.866   1200.0\n",
       "Macro avg      0.892   0.869  0.880   1200.0"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.estimate_human_precision_recall_f1(dev).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>support</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>positive</th>\n",
       "      <td>0.886</td>\n",
       "      <td>0.871</td>\n",
       "      <td>0.878</td>\n",
       "      <td>1200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>negative</th>\n",
       "      <td>0.932</td>\n",
       "      <td>0.857</td>\n",
       "      <td>0.893</td>\n",
       "      <td>1200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>neutral</th>\n",
       "      <td>0.861</td>\n",
       "      <td>0.878</td>\n",
       "      <td>0.869</td>\n",
       "      <td>1200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Macro avg</th>\n",
       "      <td>0.893</td>\n",
       "      <td>0.869</td>\n",
       "      <td>0.880</td>\n",
       "      <td>1200.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           precision  recall     F1  support\n",
       "positive       0.886   0.871  0.878   1200.0\n",
       "negative       0.932   0.857  0.893   1200.0\n",
       "neutral        0.861   0.878  0.869   1200.0\n",
       "Macro avg      0.893   0.869  0.880   1200.0"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.estimate_human_precision_recall_f1(test).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "agr_df = utils.get_worker_agreement(dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "614"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agr_df[agr_df.rate == 1.0].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1280"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agr_df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5109375"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Percentage of workers with 95% agreement or more:\n",
    "\n",
    "agr_df[agr_df.rate >= 0.95].shape[0] / agr_df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAEDCAYAAAC8vCafAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOy9ebRlVX3v+5lzrmb3+/Rt1am+o4qiUVQUUVDBGNtgmpfcGI1YxjyRF43e5EUH7+GImncl3qA396Y0MQkmhAQTdGhULtjQSRBEqmgLqq86dfpzdrv2XmvNOd8f69SBkkbIUM6pc9ZnDAZjr7XX2nPV2fs75q8X1lpLSkpKSspzIhd7ASkpKSmnA6lYpqSkpDwPUrFMSUlJeR6kYpmSkpLyPEjFMiUlJeV5kIplSkpKyvPAWewFLBZCiMVeQkpKyhLk2bIpV6xYwrP/o6wkJicn6e3tXexlpCwB0u/Cc2+iUjM8JSUl5XmQimVKSkrK8yAVy5SUlJTnwaKKZbPZ5Morr0RKyfe///3FXEpKSkrKc7JoYnnnnXdyzjnncNttt/2nAi233HILr3zlK9mxYwebN2/m3e9+NxMTE7+AlaakpKQsolheddVVfPGLX+TKK698wdfefPPNvPGNb+T3fu/3ePDBB9mzZw9jY2O89rWvpdFo/AJWm5KSstJZNLH89re/zYUXXvifuvaKK67g/PPP513vehcAmUyGv/iLv+CRRx7h2muv/XkuMyUlJQVYRLF0nP9ciud9993Hvn37uOSSS045vmXLFtauXcv111//81heSkpKyimcdknp999/PwAbNmx42rmNGzfyve99jzAM8TzvxV7aolI7so+5v9sNDz3AoVDTMBaKnSAF1GvQDsBxQM7/ydtNQEK7Bb4HrgfKhagFrTY4HiggNiAsQ1LRKQFXwstfTfe7dpEbGFnEJ055NvZP3s/dD19HM2xw4sAcdVvF0AZcIAQagAf4gMLFQeLSpkGWPC4FQhq0aeKRwcEnoEGRDpTr0tWfx/Vjugr9bBu5mLX955FxCov5yC8Kp51YTk5OAlAqlZ52rlQqobVmZmaGgYGBF3tpi0btyD7mPvCbC68/cc/j/GCq+nP9jM/sGOE3R3pBAz/4NtM/+A789VdTwVxi7J+8n6//5A8XXn/qwz+g3dQ/18/46F+9jMF1Bar1/Rx6+C5WHXkZbznvT5a9YJ52Ynkycv6fqe3evXs3u3fvXnh9UnhPd1rfu/Vpx9blfP7vratOOfa+H+9/3vf84rnJzn02ivnY3sPP8A7L9A/voHHBG17QWlN+sfzkwE1PO7b9FT287NLBU459+f/d+7zv+Z6rzgTg0MMVvvcvR552frZ+iKOj++jIrn6Bqz29OO3E8mTtaqVSedq5arWKUorOzs5nvHbXrl3s2rULSMR2udTB1i56HXP//NenHOvwHC4d6Djl2JE3veQF33usFT7LGUH3+ReQWyb/hsuFs3k7R35yxynHeoaynPmqU/9Of37zxS/43kY/c4pfZ2Etq4c2pzvLpca5554LwP79T98lPfHEE2zbtg3f91/sZS0qheENhB+5muY3vwoH9oHng4hh6zmJz7I6B0ETXPdJn2WrDqjkeNYHL5Ocq81BownWJv7JVpS838mA9MF14BWpz3KpsqH3HN569mf54YN/Rz2uAoLEP+mR/NzbJL4UD8gCCg+PkBCoAx4uXcQEWMy8DzNDgypFfOBBCmykRD71WS41Tpw4QX9/P1Imgftzzz2XTZs2ccstt/CJT3xi4X2PP/44hw4d4lOf+tRiLXXRsFFIvneA4u/9IaYd4H/4v+JVqgz/6bUI10Mo9azXToyP0VMuI1wPG4WEjz+CzGSJKzPYKCSKLHz3dRTechndb3szMpvH37LjOe+Zsris6d5Oc9MltKIqSlzP5oFX8Y5zPoi1lpzfQRg1Ew1F4EiPrsJqJqr7qVaqtOUk/aVNWGvpL2/CVdmF+95YuZEv8C1+5ZV/zI4dOxbt+RaLJV0bfsMNNzA0NMQVV1yxcEwIwRe+8AXuvPNOvvKVrwDQarW48sor2bp1Kx/60IcWa7mLhnA9hOcSN2rYVgAWTCugtfc+Wnvvw4RPN6Wt1uhGHf34I7Qf+gntfQ+CVOAo4rkZkApV6MBGybV6Zho9MU506HFMK3ixHzHlBRDGDSLdIucn7qgwaqKkj+fkieIWkPj+HeXTaM0y2zyBtZbYxAgraIcNLAJjNcY+GRwy1sz/Pzke6eCU88udRdtZXnvttfzN3/wNMzMzAFx++eUUCgU+/vGP8853vhOAjo4OisUig4OnOqcvueQSvvWtb3HVVVfxmc98hna7zStf+Uq+/OUvk8/nX/RnWWyEUjgjG2nf+g1su4WeGue+vQ/ya+//fT5w8au56Lcs2Z0vBZJdqIk10aMPoKMY/ZO7iddvwlpwVq2F2BAfPUiz1ebfHt3PX371a0/5IJj/vaQsYTwnjyM9mkHy25JSIYWkv7wJbUKMNYzP7ePA+F2Mzj6E7xUpZQfI2yFacppGe4qc30WsG3hOgcGObQDMNo4DMFHdz+gcRLqNp3IMdmxDiuVvaSyaWH7oQx/6mbvASy+9lGr1mVNg3vCGN/CGN6SR2JPYdpDsAltNPvf2N/KlYpa/v3cP37x/L+f++3f5vz7yEd5+9g6kNbQfuAcbtrCOh61XiRt1VDaHDUNOjB7jf/3dV/ibW29nqtFk+5oR/udVH+cdO7eBn8Hp7kNmsj97QSmLSinbT6AyCCFQ0kGbGG1CpmqHabZmeGTsVoKwSqM9TT7bTSOYYbT5GC09DVhyfjfre16O6+boKa4FoBqMATBVO8TYbIxSLghJT3EtfuqzTDldEH4iYLo6x/DqYf7k19/Bh9/2S1x7y+38+be/y7suf98Lul9Hxue637mMX3rd6/A3bqP9wD3oVjMR5a1npj7LJYqxmuMze5moPo6rfKy1NFozzDQO01kYZrzyGJPV/cw2j5J1O4h1yGz9CFm/i0g3EEIQm5Bmu0K1NYYbZQmiOmAxJj75KVTbEyjlIKxcMaZ4KpbLAKs18ZEncEY2gDGInn5+/J1v8Fffv5ub7vkxUkqMeWH281yrzR/c+C0eOHKC3zl3O93S4m3dAQ2PeGYSt3cgFcwlSKQDputHmKodItRNwHL/PY+x5/5HyJ3XwWR1P0FUJYwDwjjACoHrFFnTuZP97Z8w2z6AsRohXUp+L82oxhNjP2D/wf186xvfB6CcHaDouzjKRaBWhAkOqVguC2wUYlotpOeBcrjyk3/K3956GwXf43dfdha7Lnkt61athnwJ5SicdZtpPfITdL1C+8QY+YFB1NBqZDaPCepYY7lt3wH+xw038tnv3cXnb/8P/uqXXs2l2TzaH0UNrMLMTKVR8SWKNiGxCRHAr73nIv71utv5nbf/MS991Vbe8e5z2bCzl3JuEGM1GdWBoySxjVlVOptslCXvd1FrT+K6ecYOHOErX/xHvn3TXQgh+LX/8haGVw2jVQWLpre09pSI+XImFcvlgFTE48cx1Srx9DhHpmbY2NPFN9/zq5TyWZTnYmo1nHInzuYzoN1C6BivdxXtTJHsxW/EnDhGfPwoOA7RiYNcMNjNK97/W9y77wC//LndTAkH2dGFrcwSPfxj7OAI7poNqPzy91WdTijpIYUkiCoo4fCuD/wyV175Yb7y5Rv50v/8B/7kfY++4Hs6ruQ1b9nEb73vTVz4ksswOsJzh4niFt2FtenOMuU0wmic/lXQZ4lnp7Fhm5LnUCDCNmKiVhOvqwfheshYY3SM09mLGlkHe39MdN8PsVIRz4wTV2tE+x/BHVkPUcjISJJ4rnr6EY6DiULi2RlU9+DPWFTKYmCtppTppzO7GikUpWw/W1a9nN99dw8PPfIA3/63/3jB91y/vYdL33Y+G9duZbZ2mLHaflpRhZ7CBjJeB8OdZ6wIwUzFchkgXA+Z8TGNOu7gEKrUiWiHeJt3YB0XM3YM2dEDcYiOQ5yOHsLjR4kfegAmxtA6RnX1gJ/DVA5h20GSnN7VjfWSVCwhJUiJHBxCCoW7dkMaFV+CWGCs+hituIoUktrkEB/80yu57u//AYvh9W8/i8vecwHlPpeewgiOk+GcNW+n2hzn0SO3Mx0+QVd+FcLAzNws13/5m/zvGx/hv777b3n5q+/lv3zgUrafvQZjNYVMF1q30CZErgBTPBXLZYBQCm/jGUmyuONigiamUSf40R1JOWNPH2ZuFv+slyfiOH4CMzWObgUgFbpZRxTL6Oos8dQYCAmOi7d6PVSTGnwrFWZ6EpHNIcqdeJtTf+VSJJ5PRi/l+pmuHeKtr3kPrVabt/7aa7jsPRfS099BFDeITZuMV8AYzVRlP0dn99KK6sQ6oNmu4KkcqwY38eu7LuStv30+N//zT/jWDQ9wxW8+yKe/8mvs2L4dR3i4Tg4lV0Y7xFQslxEmDImmpzCtBhgDYQSNJqpXYWpzCGOwUQuRK2KlxE6NQ72O3bAF2T+MaDYXhFIWSrg7X4o68DgA8fGjxE4LZ2gVJmxhgjoUUn/lUsNz8rgqQxBWkFKhteb1v7KD9370Ytq6Ra09hiNdojjGWEMYtWjHTarBCUIT0pldxVDXmZT8PpTjMll9As+JeOvvvJT+1R18/uPfoej3cOaqN9NZGCbnda4IExxSsVwWWK1pP7yH5t23Eo+NYqP55hftACToRhXleeB56JkZZBShJ8cRuRwoB3dgFcQR9ugTSARGx7ir1uLmsmTPS0Z/iFweE7WJjx9B1SpEhw/gdvWmu8slhhSKzvwaMs4sMnAAi6M82nFA1i3iqAyV5jizrUME8TRhHBCbBmHcBDwMFqzB93JM1g7hKp9mWEWbNrFOSiWDsM6ByR/S0VhFX2lDWsGTcvpgoxBdr2AtCD+T+BddF9Hdh81kcLr6cIfX4HT3YDwXZ3gNGJuI7OOPoqfHscKgOnoQ0iGemcIbWYv0c5xsGyocCY4C10V2dIGOsVGIWAG+qtMJbULAUM4P4KoMIMi5HQx1bqO3uIGDk/cQmSbtdoA1Gm1jOgsjuE6OuC3o715Ld26Ecm6YenuavuJ6fCdPbGJ6igb4AYMdWxFSAYZIB6nPMuX0QbgeMldEVyoQhiAlQrnIjI/WBlOvEU+MIpQDzRrR0YNJ04xyF/QNIDu6EQMjyFgjOroQ5U5k/1BSCO64yWd4GVAgHBczN421AuGuDF/V6YSSHo70CKMmc8EJwBLGTZTwKedXUawdJIgqTNcPYY0AIWnHdTwnh9IKYQVCOsw1TjA2+xizwShh2CQyTWabSRPoZnsGa0pYq3GUl/osU04fkgDPFvKtN2CNRv7zt6A+hW00oBlgmMCMHsaMjSIGh5FBE4TFTIzB3BzB/seQe3+Eu/0l2OocsqefcO+P0V29BPXGwmcIz8dK0NrAQulbylJCCkV/eTPNcJbjlQexWCITcGxuL5lMkUY4g6cyOMJHSIsvCijh0lfeRL3aIJctkvFKjM89RiOcRVgQUtGdG8FTowCEuk3OLVPODiNWkISsnCdd5shMFqejE9NqJ35EJVHlLkw0Ac3E16Snx3AyGUShgI1idHUGGnWM54IxOK5LrDVSa+LKHLTbhHO15Np6FZErIDJZZCaHFTI1w5coUihclcERyc/bWgvW4qkc2hq0NnQUBgnjJp351bgyw6MnbiNo1hEzmq7CCNXWOPXWODmvC2tj2lHjyRZtxNTCaYbcM7FWp2Z4yunFyfQh3awjy52IRgNv50uQU+OEB5/Atpo4uSJy4xmoRhUdtqEyC1ZjgxgjBOHYKDKXwyqFrsxg2jnC0WTmihQSObIJZWPI5XGy2dQMX8JYwPc6EUhyXicFv5vZ2mEkkp7yOty6SxDX8VWOyLRwrIOjPFpmhlprHFd6SOGBNQx2bGOwtJknOjTw7xS8biSSoD1HPtOdmuEppx9Wa5p33IqeHIMowlu7geIv/yrB3h8T3Pp1rJCodgvv7JchhCL62j8kfiurcc84i8z2c/G2bsfGmnjiBPHh/cmICSCuVdFPPAR9QyjAWb8ljYQvUUIdcPcT13F46j6Mjam2prBW4zo5OlSOcnYAhYMhJjQt6q1pquEYzXaNot9DO67TNAZPZShmesl5ZVq6QX6+mXDGKyGlQyUYw3PyGKvTaHjK6YVp1jH1KlbrpBPRsSNkd55HdusOzLFDyFwW245wfB8TtnFKRULXxWm3cUtlTHUGfeQQxlqsnyGu1yBKfJMik0EUSvgja7A6xjbrUCov8hOnPBON9jQztYO0dR0LxDqg0pog73XR37GVvvIG+ju3Mj73GHONY0y0Z+kvbWGmOs7agXOYrR2k1pqmrev4TpGs34krM8j5HWTGK1H0u2hEs0zV96OUw3DnmcteMFOxXEYIP5vs9oxGCIGYH9wmcwVsq0k8O4XI5nC3noWJAuSee+HRhyCXAdfDxIboxFHae+8nbjagWgU9n7PZaiFzOUwYoQpFVLlrEZ805bnwnByOk8UYzck+lNX2BKWwh0wwthDBVsrFYvGcHJXmONq2CIIptI0pZ/sIYp/QtJisPo61lkOTdwMwOvsQpf4NBHGVzuww1rIi/JapWC4TFnpartmEzBcRbow3vBZrNBiNt3UnQggsFqzGHD2MU+qGodVk1m/G6R9Cj48SHT9OPDuF092L7R+E2VkAnMFhOt73B0gLqrsflV3eP4zTmaxbZueqt6JQCG5CSheJIu93EsVNlHSwVtNX2ohEUc4McGDqPyB2OV55FE95SOXSV1yPNRalXKrBOEFcB0DrmOGus2i0J8n7PbjKXxF+y1QslwmmFaDrNZxCCZHLI6Iq0YkjRCeOYoWAKCQeH8XpGUA4LtYCYQsadezMFGrjVszMJGidRLujCGf9BrzZpDZcZvOYiXGsNdhmHZXWhi9ZpFAMdm6hGU4jhCTj5hFYqsE0Ga9AqFtknAK+U2C460zq7WmOVvZQjw/R1hX6S1vwnTwFr5ejsz8h1AGtqIYzLxdKCjzlUyxvo6eY9LNc7iY4pGK5LLBaEx3ZT3jsCCJqg7HzJwzEBhs0k1LH8RPYOMZagxocThLO127AWbsJ2dOHE4YIa7GuiyqWUIUS2ZEMAKJQpHXfXQipUP1DuCNpL8uljJIeGbeMQODKDIPFbXSV1mBNTDuqU84NEkQVHOnhKp+C303RG8AxLmCwWKTykSiGO89gpnaEUibJuV3f80r6O7biSHfFCCWkYrksSDqlt7FzM+jqLCZoJDPAjx/FzE4hOzqx2qDbLczR/cQzU6iJ0WQnOTtLa3wUPXUCU6uiCiWUlPibtxMfO4KpTgGgJ8fRZQcxn7Nn9cqYu3I6YqxmvLKP2eAYNnG8AKBNm3bU5ETtUfaN/QApXIY6zqCnODIveBIlFO2oRTOcY6ZxjFg3qbTGEEIQzteGIywHJn5IOTuIqzL0lzevCMFc0nPDU54fwvUQOsI2akmkut3Cao3TN4AqlHEGVyOLHVilkvpxJSBo4XQPQiaLiCPsxDimUUd196GlJdq/j3BqjPjoIQDi6XFM0MRKiUrn7yxpIh3QDOfQug0WIhNQj2aITYQjPbACgSQyAe2oxljlMYyJKGX6Kfp9lPOJCHqOx0B5K1IIHJGhrZNJq3PBcYKwghACbaL5evTlTyqWywChFN72c1EDQ5hmM2nPpmNsFCNLZczECXRlDuG4ScK6n4d8DissuB5WOVjXQ4+N0tpzH/rIAQwW02qh5zsYCW1QvQOoYgfumvVp498lirGaqdoh6q0J5oIxLBZtIoyN6C6upbe8gYLfTTOqUg8mGas9QSOcIeOVEMLB9/IE7ZmkQYaOsdaS8co4ykWbxJpohnPU2zNM1w4jhFwRwR1IzfBlg8pmyb3ul8FoZKGIqDfw1m0AK4iPHEA4HsJ30GNjOHkHZ2gts2s38n9c/n7+4U//H3odiTUxbt8qGuPHiB5/BNOqI8t9yQdIicxlcTp78DZtS3eWSxRtQrSJcFQ2qbQRkoLfTWd+FRmnwHDnmfQU1/PQ0W8RRDWkdCn4XWS8Ml3ZHGuGd/DY8VuQ0sEYy5q+8zgxs4d2VKeUPQpAwe9l08Br0HFAT3HlzOBJd5bLBKs11mic3iFwPITvkznzPPxtO3HWbcJMjWFrDYQxyNXrMY0qn/7z/85/3Hsf/98N/4q7dTvO4Cps1MYpdSI7u1AdPeDMR0AHViUO/8ERbBhiwpVhep1uKOlhMQThHFm/hBCSYqaP1Z3n0lUYQZsQ38mR87vw3TxYg9ExQWuWucYxpmpHkUIhpUJKgSMdOvOrGezcTkdmGICS34c1Mb5bWDGTHSHdWS4LrNYED/+E+rf+DcJW0kLNzdB+9AGwSQAIHSFyWeLD+2nc+nUmpcvf/+M/Ya3l7/7+Oq48Yw3D6zciVq/B8XzM8SPEtTmMk0S89dgxwiOdROPHMKNHkIUC+de+KRm/m7KkEDg02jPQTl5Loai1Jxg99CClzAC9pbX0FDdQzvQzVT/I6NwjjFX2EegqD0/lKWR66C1sJDJthFAooZipH+HxydsBeGj0O2zZup6hFTKo7CTpznIZYKMQOzMFOkZmc6A1tFu0/uMHBHf/AD05DsYmCeW9/Tjd/Vz7wOOY+RQjYwyf/fq3QSU7isyqNWTOOQ+nsx+VSaqAMBrV0QXNJtL1sK0WpllfxKdOeSYSM7xNZ2GEjtwQIDAYBIBIChJaYZ2e4ggDnVsp5geITRtjIwQCoRQGjefmyWe7UELhu0VQAiWTvVVbNzhRfXTFBHZOsmhiGUURV199Ndu2bWPnzp3s2LGDz3/+80k7qedx7Wc/+1l27tzJzp072b59O2984xu54447XoSVLz2E6yG6ekA5mHYLlMQyX/6YyWLbAc7wCE7/IKqnD5HPc+/BI4Tz6T+h1txz4AgIgcqXENkc8eQ4JmgQzY4DYKIIPTuFMTE6DBGZDDKX5lkuNZT08Jw8woIUDgKRRMCFBAPV1gS11iQz9aM4KkM7bOCqLAKFQIC2ZJwCGVVAGIh1O0lA97oXPkMKF0/laIZzxCtIMBfNDH/ve9/Lbbfdxu23387q1avZs2cPF1xwAZOTk1x99dXPee2HPvQhvvzlL/O9732P888/H601H/3oR7nooou48847ednLXvYiPcXSQChF9oyzcUfWYRoN3O/+GDkxhuodAATuqjW4G88geuJhnNXrsUJw92suBdejWqvSc855SU8vrVHFMqYdED3xCJkzz8F58AEA3JH1FN/4Kxgh8VavwenqTU3wJYgUisGOrfQU1wAghCTvd7Gp/1XEJmS2foSMV0briDBuUMr0snPkLXSIrfzBrk/xZ3/5B5Q7MwilcJxOIt3m6OxeSple1ve9EriVrYMXE8QVDkzcjefk2Db0+kSQlzmLsrO86667uO666/jEJz7B6tWrAdi5cycf/OAH+cxnPsORI0ee8/qvfOUrXHLJJZx//vkAKKX4+Mc/ThzHXH/99b/w9S9FhFK45S78odUIx0Fm8+QvehP5i36JzJkvQToKoRycjk6IQqyOk9JIx0U6Hmb8OHr0KNHBxxLhdJykGYc734wjk0FkMrilIm7vQCqUS5ik+e+TlTVBWGG2cZxGaxbPKaB1hBCS2cYo1dYk1eYJvvT5m/jJfQ/xV9f+E23TQKGYa47SiKZptGcQQuLNB3OUdIh1hOv4RHFAGDcW83FfNBZFLE8K2iWXXHLK8UsvvZQoirjxxhuf8/pSqUR0coLhPCdf+/OddlYiVmt0o46NImwcEdcrC5U2wvUwsaZ571009/yI1p77CPb+GB3HxJVZdLOJzGSxUYTRIbbVIpwcRzeSRGThZ5H9g6ihEUwrSCt4ljCxCTk+s5fjMw9irCbUTazVBO0KGa9EOT9Ezu/EmIiu/CrGxsf4l3+8CWst37jxB4ydGCPUbRyVxRUZtI1oxQ1ik/zGpJA0wlnGZh6lGkwgpbvIT/zisChm+P3334/neaxateqU4xs3blw4/1x88pOf5Pd///f52te+xtve9jaazSYf+9jH6Ojo4PLLL/+FrXspY8KQ9sM/ITz4ONGxg0S1BjN/8Um8dZvJv+IinHWbaN13B+GhA5h6BdU3iInamOOHaU1PYOoV2HkewnUJbruF4OEHiI4eIqwmYyXCfQ9R++Y/43QPolyFu3Yz/pa0mcZSw1jNibmHGa/uY7J+EGNjJiv7uf/I14niBp251bTjBr3FdTTCWcYrj/M//vQm4vlWfFrHfOV/fYc//+//DdfxmKjuJzQh9WCa0ZkHAXjkxK2syfaRKazGEDNeeYzVXWcv+8j4oojl5OQkxWIRcXLO6jylUmnh/HPxu7/7uziOw+WXX86uXbuYnZ3lggsu4L777mP9+vXPet3u3bvZvXv3KetYDlijMY/tRR98HDszhYk1WEPcjogrFdqjRxFaE89VIZeDiRPoRh0yOWi3aIQhRjq0XA8QxDOTSdPfsA1usmuIlSKYmEQ4WVQ2B2MnUKUuhJ9Z3IdPOYVYt5ipz1ALalRrk4lLBYEOLdYqglaTZjhDyVlNO2zTCuscfHgGHSeB1TgyPHz/EWrVGnm/H9N2cU2ZavsEYRQkH6JdYg2tVgtP5ZmZmcaLRnHU8v4uLIpYWmufJpQvhPe///189atf5V//9V+58MILaTQafOQjH+GSSy7h61//OmecccYzXrdr1y527doFgBCC3t7e//QalhKmFdAudxIPDBMGTaRSICXKc/HyefKDQzhrN1HrLGNaHmbdZrwNmxGOS/XhB/ArEwg/T8ZqiDUtBFH/ANHcBGI+2On6WXL9/ahyB8p1cAcG8QeH0p3lEiM2IVPxXgqqQEOXEELgKJ/OYh/NaI6smwMVk8m55PKraNlZPvHFSwjjAEcVGO7cxnDndoRsUbP7aTKBdQ0lp4Tv5QHIeDkGOtZS8Ptw3Sx9xSH6O4bSneUvgt7eXg4fPvw00axWqwvnn42bb76Z3bt386lPfYoLL7wQgHw+z7XXXsvw8DDve9/7uPPOO3+xD7DEEK6H8FxkZxd+8SU4Q6txqlWKb/l1pFQgFfr4IfyzX45oB6itZ2KbDdAaUeogWy4T16qYeg23WMbdfnwAyxgAACAASURBVA5OrUp2x7k0pqZh9z+RPe98Cr/869iwhSqUcIrlVCiXINZqytlByrlBMCCEoqewhpdu+HUi3U6GlgmFtRbX9Slnh1ndvZNSZhjddOjoSebsVBpHCaIa5dwQvsoz1LWdY6vzwA1s6Hs5a3s3Y7EMlLeSdUvLXihhkcTy3HPP5a677uLYsWML0XCAJ554AoCzzz77Wa896c/cunXrKcc9z2P9+vXcc889GGOQcmXl21ttiI8dRk9PYWoViGOcfAGVKxBX5wBwSx3EDUXrju+ix46DlNh8keDw4xBF6HoNb+1GTGUG1dGDKpdRpY7k/nMVGt+4AWKNM7KWwsVvTsVyCaKkh5Iu49XHaETTgAUhmasfI7YhtfYU/eXNtMMakW7TjGepBhNorZmaO44bQDusk/e7ktxMBOXsII3WFJ7KAeA6WYKwQjtu4Mksw11nLuozv1gsiqL8xm/8BgC33HLLKcdvvvlmXNflne9858KxEydOYIxZeD0wMADAoUOHTrnWGMORI0fo7u5eeUIZhdBuIR0n+X5bYL7vZFydQ3oZRC6HaQeIOIJWgPR9hADhZVC5It6aDTiFfNLWraMHkc1i6jWIEjvctAOEsUg/g23U0+qdJYoUip7iWjqyw/QU1gHMdx3SZNwiAkE7rOE6OZRwaYe1hTETlfZxmmGFZlQl53XRX9zMcOcZDHRsxpiYSCeTPoUQtOMGvpOfv//KSExfFFV51atexW//9m9z9dVXc+zYMQD27t3LF77wBf7oj/6INWuShNobbriBoaEhrrjiioVrL7vsMtatW8c111zDo48+CiRCedVVVzE2NsaHP/zhF/+BFhnheohcDisVVkgQgJTgukmZmyPxNmzD27gNb/u5UCigm8nkP9Hbi+zsxLRbiGwBkS8iiwVU/xC4zkIjDeH7WCkxYQuRL6TVO0sYJT1cJ0Mh0wOAlIpqa5w4btNTXMdg5zYGO7Yy3HUmvaWNFDJ9SKFwRAYpJIJEEHN+maxXJo7bzAWjVIMkINpdWEtvYT2lTD/OCpm/A4tYwfPXf/3XfOpTn+L1r389nuehteaTn/zkKcLY0dFBsVhkcHBw4VihUODuu+/mz/7sz7jsssuQUhJFEX19ffzLv/zLKbvSlYJQCn/zDtyRDZgwRP3DTYhqLdkJljow7SBp3ZbJYrXGX70BVe5CZXJEQ6spdPdimvWkPFIAUmGadSLHRVWTHaTTO0Dpne/CxiGqUE6T0pcoxmomq/ux1mKNQSBxlU85O0R3cQ2Zn/IvDpS30FfaBFh820cmp1DKQQkXIRQCh1K+l9i0cNR8ZoRpM9i5E2s1Snorwl8JiyiWruty1VVXcdVVVz3rey699NKFoM9T6evr45prruGaa675RS7xtEIohcxkiY8dxLZa2LANjkrMZ9dFuIm42SgELF7fYHJOG6TnIb2un7pfGZ3xMVPJWAnhejjpnPAljzYhsQnx3TxhYlegTYyr/FOEMsnHfIRQN/FUjsGObfQVt9HVXcJYzUR1P56TpR01masfoxJMMNc4AcBcY5TJ6v4VM07iJGmLtmWCDgLiiVHCuQrWGGwcYb0MTk8vKpuY3SYIsAqiyixMjCG7e4lbdVrHDiIcD9OoY+IIt6cfp1hGDY4gWkmfr7gyQ1StQBQiPA+VK6QBniVIYhILasFk0lBDCFyVpbOwmnZcpx03MdYSRHUOTtxDxs0nMSAh0bqIq7K04jqtdhWjI4SURLpFOTe8UP9traEWTJDPdFPwu1eMYKZiuQzQQUD1n/8GXa0SHnyMeHKMhw8f4zUXXYzwfUS+gK3XkjJIHYPRICTCcTHWILTGxnEyDRIQno8slsFaWrVkZ9/47jeZrh7DGR5Behmyr7iQzNadqWAuMYzVzDaO0AqrBHEVbWIma09w9+PXYdBM1Q7SCmtEuok2IRmvgyhuc3TuftAZyl3/Jw8c/jph3EBJl5Huczg6u4d6a4KJ6uMA7D32Leb8HvrmHmJD/6sZXiF9LVOxXAboygw2aKDyWTCWM1cNosMWwvcQUiCFwLhuMunPahAK4ThYC7cfOcGr1wxhjQYrQQqEEkhhwRp8R/LqoV4GCnlMrYacb6FnaxVsFCJWUKfs04EwbhCbkGymTGX2OGARSJrtaYRUWDSaCI3GdbNYYZESsn4XtcYs09WDRLpJPtNJPZimGc1RyvYS6xauk1ToKOmS9ctYYWmHVbQJkSvge5CK5TJAlbvAzxKdOILF8scXvxJ7zmac3j5U3xDejpcQ7rmXeGoUXU3SflQuh+rsof+PPs0//eovJXmXURuEQK7dQO7cC4gnR4mOHERPjSE6upCZHAaB1DHkiwt+0JSlg+fkcVWGKArw3RIgCOMmsbEIIrASaRUKFyU9yn4f7TikHTZwhE93aR3HZvcStCt4Tp6C1810WMdRPlIkcuE6XtIvE4XvldJoeMrpg8pmKf7Kb9H+8T2QL2JbDdSq1QjpoCfGIY6RPf3kNm9H5PI4w6sQlmTGzh99mvwlbyc6cQwhJKKjk9xLXo5T6qL+v7+GDUOcgVVkznoJ/lnnoZ94FGMtUqVfnaWIIz22DF5EEM7hu0WU/EM6C6s4Z+1babZmiIkRQiIRdBZGKPhdCKFotmcIag4lv5eXb/jNheuna4fpzK2iI7uK7nwS7NvQcwGv2PJafCeH7xRWhAkOqVguG6SXxTRq2Nmkgkfmi0THj6BPHEsqdYzB3bgVpMDr7j9llK3T0YG0GtsK8Nasw+8fxrQChAUnk0X4PqpQRkmFzeZxMllMO0jN8CWIsZrp2iFiE+K05wDB2PEZvvHNm9AmEcquwiq0ienM1eenOMZI6VCt1Ojo6Fi4V6zbzDaPo6TLdO0w9z+QdB1ylEve61hRw8ogFctlgdWa6PGHIIqQjgOFMkoI4lYLpIBYg7KYsIUqlp5MI5rvSaknxgCDO7Ie7ylBG+G5CD+LjdsIa5I8TM99WjpSytLhZOqQ52SJ4hYA3/7GrXz7G7f+XD9npZjeTyUVy2WAjUKsBVkoYpo1hO9jpIR8HmansTrEWbUBb8sO1HzljWkF6HbScstduzEJEA2PJPeaz9n01m9B9w2ix0fB9YmeeBQ5MIjKZNPUoSWKkh6O9IjiFkJI/v3bX8dayVzjOMbGCCTl3ABSusQ6YK55As/JEuuQt7/xXdxx5+1M1w4RmTau9OksjKBNm6PTD6BNiEDx0rMuWjGm91NJxXIZIFwPmfGhfxCHQdxN2xEC3HUbaX73myDmR0TI5AsePvEwutGkteceAIK7v4fqX018Immu4a3fhL95B/6WHehahcjPIjyP5g+/jyp1IEsl8q99UyqWSxApFP3lzUQ6YKp2mDVbO3GkR2/pFQsVNwDjlX1EukUl0HRkh3BUMmHgpS87m0dGZ5FCYK1l69C5SKE4MdeDkg7GaPK53GI+4qKRiuUyQCiFt/GMxIfoegsiZrVGeFmEUgsmt41CTKuFaTWx813QZbaYpBlFUeLfDJrYKERmsqhiUskTz85AHKM6ujDNJKL+01U/KUsDKRRSKIyNF6pwwrixUMET6WChyqfMk2WQT5J0KmI+TUxJD1dl0CZaUbXgP00qlssEodRCsMVqvSCc3vpN2KCJyOZAKuJahfbRg8RHDxKOJoPh4qCGW88Sz06jevuxCMxThNddtwXROUM8M5nUkKdjcJc8J83xdtRkrnmM2LTwnQKDHdtOOSdEkm500qx2VZbuwjraYRXPffJv3F/ejDbhiqoF/2lSsVxmWK0Jn3gYG0YIz8XbsA3TDtDNOvVbv4mtzxFOTyGlxJmPfDrFbvytZ6Krc0lbt6OHiI7sx9t4Bv7m7UQHH8O0WrgDq3BWr8Upd6WNNJY4J83xZjjLdP0w9dYUNTtJT3EtvlOgt7SBE7MPY61dqPN+8lqBVIq55lHAoKRHT3HNKRMjVyKpWC4zbBRiwwg5n95j2gHBD7+Hnpqi9dB9eOu3QmUam8kj5nMlhaswzQYqmyMOmujZSWxsiMQjOIOrMK0WenIM02gkPTO7lsc4juVOMhI3gxACi13oawnMV3IJfCdHFLcWelJqE2KsxlVZwqgBwjJVO0CoG3gqR09x7YoVzZXVJXcFcHLExMn0HhuG2CBAljvBWGyrieodxH/ZBYlwAu6Grfhbz8TbuhPlZzBhGzAYJMLzEEIkQpnLJ32Fo5XR7HU5cNKs1jpECMVM/ShmPtBzMmqupLvgh0zMbIfp2iEaUYWJ6gGM1fhOnqnaQUZnH2a8sm9BdFcS6c5ymfHTwR6rNSKbxQZN3M3b8DduR3V2423YSvjYXgCk6yAzSSDI23Im0dGkzZvwPaSfxdu6E0gasEvfT/MrTyOkUPSV1hGbgIxbROsIbUJclX1GP2TSaX0NoW7QW95IGNaRyiGMm1gsGa+E1uGKqQd/KqlYLkOeGuwRSpF/7ZtOae4rXG+++mb+z38yIKSS8+7IeoTjJp2I5psG+9vPeVq0PeX0wFVZMk4ilE/dRUqhnlHwFt5vIjJeid7ShnmBzaF1eMo9VhKpWC4TTBgSV2awWqPyhaQ/ZbMOSoFIktBpB6hsAWFB1yvEjSR1KBofRwwOocIQo0N0s4lFgxWY44eg0YBCEakE0s/jdvWmAZ4lTKgDKsE4kW5jEQgMQrpk3AJSCOrtaXJeEtxrRVUiE9OOku9CK65TCSaIbYzrFlBC0ghnUULh+x3MVPaTzw8xVt1HrCN6imvIuSujKXQqlssAE4bUbvk67T33omsVnNUbEK5CRDG6MouxBlNJEo3dtRtBOpiwRevRpNZ37vq/wr1lALlmLcxVsTpEVytoY+D4YdAGdAzlTty+QbKvvJjipW9PBXMJEuqAHz7+txyevp96MAVYLJDx8ngqj+/myXkdDHfsQEjJVPUwR2fvX2iM8p0HrmG6sZ/ItMAKCn4fENNdWMuBqXsw2mBo40gPpXw6s2t487l/siIEMw3wLANMsw6VKgKBRCIaDezcXDLlMmxDrYEwGhDJZMbKDDLWMF/uKBGYIIBqHcI20oANI2g2QSdRU7RJmgabRHjT6Y5LkyCcoxnOYYUBYTBorDAYq4l0gLExyGR32WjPYmyMsRGOSObr1MMJrEj2o7GJsNJgiGnGFbSJ8F0PjcagUVIRmgaV5olFfuoXh3RnuQyQuQKUS0kyORZZyCMchYlj8HxwXWw1SvyV+SJCKkzUQviJv8pIgZvNQqkA1mB1jMxmMI6TmPHagJIgFUiJLHemSelLlKzXQc7rQFqZ9K4UEmstUjjzKT8OGEvB70ZISdCuIYWHtjEABa+PdlTDAkq4WG1BOOScTpR0CaMIhULioI3Bk3nKucHnXtQyQVg7X9O0whDzta/LhZ/pswwCECQ+Szc5Z4wm0zdIcGAfsqMDqTys1Zh6Ax21EK6DDSNsrZr6LE8jnslnqaRHxi1grUYK9Yw+y8HyZoKoRiWYoBVWODZ1P23TouB1snnotRhgtnaYTKYXHTfAWnpKa5eVCf5cupCK5QrlZEmkyuZO+XcwYUjj+/+ODQJENkv+tW9KhXGFcPI3Yazm4NQ9PHT8ZjyVwXeKvGTdZWTdEkIoJqv7k36Z0lt2Ex6fSxdSM3wFcFIYkSrxO0pFdPCxxC8J6EZ9Ic/SNOvYIEAVy8TVOeKZSdzegTRd6DTEWP2C67kTv6YGDK7yiU2Ii2GmdgSlPLAWi8V38wuVPysl3zIVy2XOyVpx02oRjx/H6V+FECR+rEzSaqv96B5UoYi38QxkroDIZomrc5jKLPHoMWxlBm/jGalgnkYYqxmv7HveO8CTFTljlX1IofBUjt7COozV9BY3IKWD52QJoyZCiKdV/qwEUrFc5pysFReOiw1aSKUwRs+XMCZjblWhlLxvvi1b/rVvIp6ZJB49hlMspSMkTkN+umP6U3eAz7TjPFkbfrKlm7WWrvwaHOkx0LktaQgct3CUT29pw0JvzOVkgv8sUrFc5pysFdfNAByJjiNUNou7bstCjXdSmfPkmAjpeThdvZiJE+hmA5RCN+ogVeq/PE14ttrvZ9txnjwfxS2sjWlHTfLZHqyJEZzaog1Ar8Da8EXLs4yiiKuvvppt27axc+dOduzYwec///nnHXSZnJzkyiuv5JxzzuGss85iZGSEiy++mFtv/fnOGjndOdmPUiqJM7gGKWXy2vNQ+ST9x9u47RQz22qd+DQtWB0THd5PcMctNL7/75gwbaJxOnCyRVt/edMpJvhTd5zaRAs7ypPnu4trmAtGOV55iP3jd2ARCzvIkwPKxiv7GKvsW3ENNRZtZ/ne976X2267jdtvv53Vq1ezZ88eLrjgAiYnJ7n66quf89rR0VFe9apX8YEPfIB7770XpRSTk5O85jWv4Uc/+hGve93rXqSnOE0wGoTEKeYx7SB5/RSeOukRnjTdVS5PPD2ODZo4XT3oejXtkH4a8Uy138+04zxplgNYa9Ampq+0nqA9R2c+yaGMdICS3oLYOsqnHdaIdIDvrIyc20URy7vuuovrrruOL33pS6xevRqAnTt38sEPfpDPfOYzXH755YyMjDzr9VdccQVbtmzhYx/72MKx3t5ebrjhBrzUTHwaP9227Wd1DTrl/cUyIl9A16tph/RlwMkd51NN6pNmOYCjMrgqQxg18Zw8QkhOzD2CsXp+ls8GpHCYqDyOBVwny2DHthXhu1wUsbz++usBuOSSS045fumll/LpT3+aG2+8kQ9/+MPPeO3o6Cg33XQTf/mXf/m0c2eeeebPf7HLgGeb0fN83+9v2p7sKHOF1Ge5DHjqjvPkPB7Pme9SBWwbej2tqMps4wQTlSeoBOP0lzeidYS1eqGF21Nbvq2E9KFF8Vnef//9eJ7HqlWrTjm+cePGhfPPxl133YUxhlKpxBVXXMHZZ5/Nxo0befOb38z3v//9X+SylzQmDIlmJolmZ9CNOiYM0Y168l8QoGsVrJ0fgduoLwwwO3ltPDeDDgJMK3hyuJnWxPUKulZJSiONPuW6lKWJsZogqjIXjFFtTSzsGk+eezKXEoRQGBNTDyYBaEZVmuEcxmqsjcl4JQTQjuoLZvuTpnx7RaUPLcrOcnJykmKxiBDilOOlUmnh/LNx+PBhAHbt2sU111zD5z73OaIo4g//8A953etex4033sg73vGOX9zilyAmDKl/9xvEhw+gq3N4m7aD5yKMwRqNnp5ClTrR1VlUdw9CuQvjboEkcNNoYKpz+Ge+FJnNQGwIDz5G69EHUaUSIpMlc9bLUblsmnO5hDFWMzr3II+P3clU/QAZr8RI19lsH34jUqiFEbhCCPrLW5is7meqfojRuYcAuHnPf8N3SxSzvXTmVtGdX01Pcf3CDB6Ayep+rE0KKROzfGV8FxZFLK21TxPK50sQJJ1yzjvvPHbt2gWA4zh87nOf46tf/Sof/ehHn1Usd+/eze7duxdeP5co///t3XmUVOWd+P/3XerW0jvQ9iJL24CAQNu2foOD/Fwi4DdOtpMYQvyGRAYkuCWHZL4JZkSlyTCcMf7hiMYQkMxABkg80ahnNMhExW84MRlkUYOyCA0dGiigt+ra7vL8/ijq2tVrNXTT3VXP6xzO6br3Prfu7a768Nxn+TzDidPShHX6FMQtnLYIsXPnQLFR/bkI08Q5G0Tz5WCfDaJqHhR/AKWhAS0v0VHTGgyC14tz/hzR5iZoEolOnqYmnEgYxfAhWkPEmhpR2kKoOSdRvL5BvmupK5Yd5Uzr3wiFmzFNC1XEOdfYQIN2HF31Emw7TZsZJG5HOXv+DAhojTYTi4UBiMRDIHRUp4lcdTToueh6Hi2NESCCZUc5Gw4mZvfYMXTzNLqWHZ+FQQmWxcXF1NXVdQqaLS0t7v7uJGuf1113Xcp2wzC49tpr2b59O+fOnWPkyJGdyi5ZssQNsIqi9Pg+w4lTUECopBQrGsHOCWCMGgkeD4ojEjVL20TzGtijitEKC1E0HaOsDG9ZOQB5xcU44TDOiJF4CwtRfT6EaRIPtRDz+9H8XpTCAnxFRWh+P0ZZuaxZDlGOsDE9ZwjGfHhsHa/Xy8iiMspKx6IqGlbjOcyWswS0Qgp9I0BVEKE2Qk4eAH4jFyFMhGbh8UFZ6Vj0do/ZjrChOYTtmGhqPiUF5bJmOZBqamrYtWsX9fX1bm84wOHDhwGorq7utuw111wDgOM4nfZpF77AXe3LZKphkPvZz2OHmkFJDBxPLh2ROEBDxCJuuyPgzgUHOi07gaoRP3IAz9gK9HGVeMZWouUVuEtSyEA5dDnCpincQGHgCgoDJUwsvZl83xVuwCspmMS51jpsO05L7AyTym6jtGASk8puAR7h9mnf52zzQbxGPipqYhXIdjr2pmdLoISL6OA5fvw4S5cuZcKECVRWVgJQW1vL73//+7TPMX/+fAB27NiRsn379u14PB7uuusud1tDQ0NK8Lv55psZNWoUe/fuTSlr2zYffPABlZWVGVNj7AvVMPCMKMZTNAItJ9cddK76EgFS8RiohpESJF2OjZZX4PZ0CzMOlo1eOBLN50PPL0Dz+7suKw0pcasNy4mT778Cj+pFU7SUgKYAI3LHUlJwNQX+chTAq+eSc2HsbKG/hDx/CZqio6o6jrBTBp537CDKJn0KlgcOHKC6upr/+I//wOPxuEFs8uTJLF68mDfeeCOt89x0000sWLCA2tpa6uvrAXj//fdZu3Yty5cvZ9y4cQBs27aN8vJyHnroIbesYRg8+eSTvP322+4QJIBVq1ZRX1/PE0880ZdbymjCtol9/AHht1+j7a3XiX60n9jHHxA/dCCRXOPCbJz4oQNED+wn+v5uYh9/iFl3BHQt7XGZ0tBh6Dl4NB+RWAtt8Saa2k6mzLTRVAOP5kMIgUfzdurJTtYci/MrAYUzLUfc8o6waWg6wIGTb3Lg5B9oaPooq4Jmnx7DH374YebNm8e//uu/kp+f77Ybzps3j0mTJvH973+fOXPmpHWuDRs2sHr1ambPno1hGNi2zapVq1ICY2FhIXl5eZSVpWZi/ta3vkVubi5r1qxhxYoVWJZFZWUlb7zxhpy9044w4ziRNlB0FBVEazPCY6DnF+HEIu7SEIrhJXZgH5hxlEAu2qgSvFOq3Md5WZscPnTVYEr5bNpi52hqO4nPyEtJpJHOY7R6oTbqCCslEQdA3A6jKgooKqYVzpoxltDH5L+VlZV88skn7uuamhree+899/VnPvMZ/vznP/fvFQ6QTEz+m8xbmQxwTjxO7K97MeuPgaLiqRiPoqhuTkt9TAWe/ELCu3dhnqjDajyL1XACfWQx/pm345tSJQPlMJWoBX6EaYXx6AHKCif32r7Y/juRTLhhOyYChaKcMgw9h2DLEYItRxKp2/IncmXRNRnVbtlvyX91vefDM2UoznCUzFsp4iaK4cFz1STMox8nguToCjzjJ6FdmKroRCOYx49gHTsCgGfiNYlOoHgcRpWilY1GxKIyLduwJxCKAPpeKUjWQGNWiEOn3iHYegiP5mNi6c1YTgzHMRM1zCzSpzbLwsJCfvWrX3W578UXX6S0tLRfLkrqu2TyC9XnT4ytDIfcZBiKrqNeqG0ma4pOuA3lQoeO6jHwXlONMXEyWumVqKqC6g/ItsphzHbiOMImYBReSI7ROVtUb501qqIhhIPlxPEbeVh2jJjZiqroBLwjuj1vpupTzfJHP/oRX/va11i/fj0333wz586d44knnmD37t389re/5YUXXhio65R60TFZhhrI7TJ5hrDtRK3y1Els5eSnZTUN3/TrMSYmhmZ17Pnu+IgvDW3d5bNM6iqvZVeSHUbReAhd8+I3CgnHmrIyU3qfFyx77rnnWL58uTuAHBIDxZ944gnuvffefr/AgZINbZZdBTgnGiF+6ACK4cVpayFw/cxefw8dH/HldMfhoac1eEw7wqnmg24HTknBRAw90OVnwXLixK02DD0HvV1Kt0wcZ9nvqzu2tbWxa9cugsEgxcXFzJw5k5ycnEu+0MspE4NlOtzAZ5ooHg++yVW9/h6SAVb1+XFiEYwJUzrlwJSGl/YdOJrqoaTgajRVT13pM4ODYnf6LVh+9NFHTJ48udP2ffv28eqrr/LAAw9QWFh48Vd6GWVrsITUGqiq6+nXLC8EWFmzzAwdg2FXveGZuuRtd3qKC33q4Ln77ru73O71etm3b1+3+6WhRdG0XmfjCNt207Ul81t2XH5CGn7ad+okl4roKgh2t/xENutTB093EXfy5Mn8+te/7nFOtzR8dNdGKYcRDW99qS321kGUjXoNlsePH+fYsWNAoq3ynXfe6RQ0HcfhxIkTKZ0+0vDVfhiSXAY3c/S0PG5H2Zwwozu9BsuNGzeycuVKN5Xarbfe2ukYIQSqqrJq1ap+v0Dp8uvrmj3S8NDX2mJXC55ls147eOrq6jh27BhCCO69917Wr1/f6RiPx0NFRQXl5eUDdqH9LdM6eJJtjGa4mfixOnBMtPIr0YSC4vWh548g3hTEPnEMdcwYNM2P3XyWnKunEz76MYrXh+bPxY5HsE814Bg6qu5BQQdDxWkOIRQHzetHyy/AU1Qs1+MZosJmM83hBvzeIkwrguVYeHU/fk8uiqLRFjtH2GwjHDmLRw8ghIOmebhqVA2nWo6gqRqgErcigENhoAzbidPY1oDtmOQHyvBqBj5Pfkquy0xwSdMdx40b52YBWrhwIbfcckv/Xp10ydzsQn/dS9vrL0HwFBeW3oOScvSCItSyscR3/z+IRkFToLAYoons2Gd/8n8hkINWXI59/BNoaQbHAsMHHi/E2iBugWOC149eUUlg1h3kzf2iDJhDTNhs5rV9a4hbIVqjZ9GVAJYTwu8torxgKpqq0xj+GyfOv4dNnMQHRU38pwi8vHtFIn8ADnE7glcPkGtcge3YnG87ioODz5NL5agZXFEwkWuunJtxAbM7feoN//GPf9zj/tdfwOQPnwAAH41JREFUf/2SLka6OG52odZWiERIZOkFLBPFtlEcB/vUCTDNxHIQcQtCIdypvZaDYlk4Tec+Daa2A5aZCJCmBdiJ75UjwLRwms66WYukoaM53IDpRPAaASwnhlBNFAVsxyRutdAWP0fcDuFgA8l2SIfk/HGBheOYWHYMBQdFUYlYjUStJlBBVVQsO44t4sTNNuJW22Dd6mXXr6s79hZMpYGheAxUfw7k5YPfD8nHCN1A6BpCU9FKx4DHQMRiYOiQm+sehq4idA9q0Ujw+xIBUVNB9yT+eXRASwRgTQGPB7VwlFxDfAgqCJThUf3EzQi66kVxDIQAXfVgePLJMUZi6LmoaEByTrhK4o8LKgaq6kHXfAhUhBD49SJ8ehE44AgHXTPQVB+GJwdDH16TUS5Fn2fwPPPMMzz33HMcPnyYeLzz2Ct7mCyVKtsse26zFD4vWiAApoPQkG2Ww0hPbZaaahA1W2mJNdIWPi3bLDvotxk8P/vZz3jyySf52te+xi9/+UuWLl0KwMmTJ9m2bRtf+cpXeP755/vnqgdYpgXL3nSXCCPbfg9S9+RnoR+D5Q033MBLL73E6NGjOyX+/eCDD1izZg2bN2++9Cu+DLLpg9FTIoxs+j1IPVMUhbgVzupxlf023dG2bUaPHu3+3N60adPc9XSkoUWYcZxoFFQVOxzBbm1GDJPmEunySOa0PNV8MGXNno7HZOtiZXAJ0x29Xi/Hjh2joqICSGRJ//jjj/v14qR+ompYp/+GEwphN55F2BZ6Xh7GhGvSPoXMZ5nZknO/u5vd03GqZHH+eISws6oW2qdgOXr0aB555BFWrFjBrFmzuOOOO1i6dCmKorB+/Xp3TW9p6BC2jRMOoY4oQUQjiFgMJ9iA035d8TTOIfNZDj99SbGWnM3T3eye9lMlY2aYhsa/gqJkVUaiPgXL+++/n40bN9LQ0MDy5ct5/fXX+cEPfgBAcXFxl7N7pMsvWQtE1TCPfowTjWGfqkdYNlpuPiIeSwzFTHMao5wrPvz0NcVacl9JwcQug2v7qZKKknjK9OqBXueYZ5KLSv6bFIlE+OMf/0g8HmfmzJl89rOfTen0GcoytWOjfS0Q4SBEYokIq/l8YiSd4UVVVYzJF5a67eH30DHoynyWw0dXmdA9vQS03r4TyZqqAE43JZrcdM2bUTXLflvdsSO/38/s2bMB2Lp1K3V1dZdyOqkftK8F2uE2EA6xwwdQFIE+dgLGVRN7zWUJXa8WiWPLNsthYiBSrKmKBqrB6eaDCASKolCcPz5jAmVveu0NF0KwevVqJkyYQFFREV/96lc5e/asu/83v/kN06ZN4+6778br9Q7oxUq9a58xSPV58YyfhF5ajjF+CgoiZYXHnnRcLRLHTivISkNDMsVaScHEfq35JdsuvZ4cFFREFvWM9xos165dyyOPPEIwGMSyLF588UW+973vcfToUa6//nq+/vWvE4lEePbZZzl69OjluGapBx2zmmuBXLTcvAs92emnW5Np2oa/njKhX6xsTgrca5tlVVUVCxcuZNmyZQgheOSRR3j66aepqakhGAzyT//0T8yfPx9V7ddp5gMuk9ssOw7xad/22PFRurvfQ3L6JHReFlcaXtr3igPd9pArioLtWJ32dyxv2onPRX8H4qHgkmbwjBs3LqUtMhaLkZOTww9/+EN+8pOfDHqQjEQiXHPNNW7OzXRlYrDsaYhPt0tFdPF7kEOFMkf7XnFV0QGBI+wue8gVReFk4187rSX+aXkNUHCElbFDhi5pBk/H1Rq9Xi9jxoy55EBpmia1tbVMmTKFqqoqpk2bxtNPP93nALZ69Wp32Yts17Gdsf04yp729eU80vCSupREmLgd7nERso6LlKWUtyMX1g/PzkXMeu0N17qoURQVFXUZKG+//Xb++7//O603XrRoETt37uSdd95hzJgx7N+/n1mzZhEMBqmtrU3rHIcPH+a5555j9uzZ7NixI60ymayndsa+tEHK9srM0b6N0aMHANFje2NX7ZFuec0PKFnZXglpBMt4PM6JEydSanxdbQM4ffp0Wm+6a9cuNm3axPr16xkzZgyQaBt98MEHWbNmDYsXL2bs2LG9nue73/0uP/zhD/nwww/Tet9Ml+zc6TK7UA/7+nIeaXjpuPAYdN9mCXS5SFlfymc00QtFUYSqqin/utqW/JeOBx98UADi+PHjKdvfeustAYgnn3yy13O8+OKLYvLkySIej4tvf/vbIo1bSdHX44cDx7KEHQkLx7K6fN2V3n4P6ZxDGnpsxxJxKyxsJ/F3M+2YaIudF6Yd6/K1EInPgu1YImq2iqjZ6pbNJj19H3qtWZaUlLh5K3sJuqxbty6tAL1nzx4Mw3AzGCVNmDDB3d+TSCTCsmXLWLduHR6PJ633zHRdDSI3j358SZ00sqNneOo41XFkXgUfN7yJaUfxaD4mlt7MoVM73ddTyme7SXwbmj4i2HoEUCjOu4qywinZV4PsRq/BsrS0lMceeyytk/3ud79L67hgMEheXp67vG5Sfn6+u78n//zP/8wNN9zAnDlz0nq/pHXr1qUE9N7eZzgRsSjOmSCKz4dobkLRfYh2r9Wck4n1d7rQ3e+h4zl7Ooc0dFh2lLPhIB7Ni2U3Ew05nG8+h0/PIWSd44RzkPOtn75uUI/j8xQAcCp4jIgVQlVUzsZPocYK0DX5N4c0guVbb72V9snSPVYI0SlQpuvw4cM8++yz7Nu3r89llyxZwpIlS4DEEIHi4uKLuoahSNg28bamxPzt/Dw8Y8ZhWlH3tVFW3m2tsLvfQ8dz9nQOaehwhA3NIWzHRFPzGZlXQYg6LDtGQBvJmNKriZ465b4uKx3r1h5Vr4kjQuT4SxiVV0pJYXmX4y2zsbbZa7AsKChI+2TpHltcXExdXV2noNnS0uLu785DDz3EP/7jP7odQ1JCV50yl9pJIzt6hqeOnTqqojGlfPaFYT856KrR6XVyoHlp4WQKY2UUF4wnYBSlBMq+ZDHKRJeUSONi1dTUsGvXLurr61OC3uHDhwGorq7uslxLSwt79uyhvr6eX//61+7248ePp5RbvXo1d95550Bd/pCVCGbJPJVGN6/7fk6Zjm34URUtJW2arhro7RaY6/ja7em24/iM/JRACR3Ha2ZPWrb2BiVYzp8/n7Vr17Jjxw4WLlzobt++fTsej4e77rrL3dbQ0EBJSQmqqpKfn8+pU6c6ne+ee+7h3//939m7d+9luf6haiA6eaTs0Jd8ltk4xhL6ed3wdN10000sWLCA2tpad92e999/n7Vr17J8+XLGjRsHwLZt2ygvL+ehhx4ajMscdoQZxw5HwIxjRyI44VDKTBxV11EUJeUf0GnbxbYnS8Nfd/O9ByqL0XAyKDVLgA0bNrB69Wpmz56NYRjYts2qVatSAmNhYSF5eXmUlZV1eY5HH32Ul19+udNjeLbWMIWA2P6/ICJtKP6cRE2y3Uwcx7I61SyDwWBGdXRJA6fjo322uaRM6cNZJibSsJrOE37zNZRADiISJnDr/0bLK+ixg0YGSykpE78TfTVgmdKloUUN5KLk5CCiUZRAIPFadtBIUr+QNcsM48TjidUcA7moRu+N8LJmKSVl6neiL2TNMouohoFqjBjsy5CkjCODZYboLht6+4znqBoiFumy1tlVZnSZLX34spy4O+gcIBxvBBQCRiGqormD0AUQiTdj6AEAQrGzKIqGqqjushSOsIlbbeiaDwWyNvuQDJYZIDm+0olGsU7/Db1kdGKxsqsmET9ygPgnh8C2sM4H0QpGoObkkHPrnW7AFLZN7OAHxD85hKIIPBVXY0yYQvzwAcxjBxFCwaiciPfqaTJgDgOWE+fAyR2YdhRN0bEdm5PNHwAwunA6I/IqOB86ge3EOdV0EJsYKom/65+PbCFsNpNjjKQwUMbI3AqawvWYdpRwvJmxI6ovzBXvPuN6ppLBMgMkM5srugcRiaJqGsI0E+Msw2EUVcOJRSEUQisbgxONJto1LzyuCzPuHocCTiSMEw7hRNpA0VFUEJFwoldddhYNeXGrDdOO4jfyaAmfJmpHUFUNhEI41ojPm891477YZdnbp36307Y/H9mCz8inOdJAInlwGKEIAkZhVs3mkcEyA7iZzaMxFL8Px7FRvd5Eb3gggHBsFN0DubnYkTDqhZ7ylPLJ4xRQ/Yn9qj8HW9gIBxR/QGZMHyYMPQeP5iMaD+Hz5OPRcghFTiOAgLeIgDGCvXUvYztxGpoO4hBHRUNx/Ph9XsJmMwHvCIr85YzIHZeoWVoRPFoABRVd99JbxvVMJHvDM8TFtlkme8Nlm2VmuZg2y6bzYQqKvFndZnlJqztmqkwLlhdLDh2SkuRn4RJXd5QkSZJksJQkSUqLDJaSJElpkMFSkiQpDXLoUJYTjo3dFgJkj7ck9UQGyywmbBvnyMeEm87IWTqS1Av5GJ4lkmMmhW1/us2MI6LhxCwdTXNn6UiS1JmsWWaBjmvzJNfiUTwGii8A0TY5S0eSeiGDZRZIzh1XfX6cWMSd461oGur4SQRyrwdkm6Uk9UQGyyzgzh2/sBZP+9qjompoObk9lJYkCWSwzAqKpmFMuKbHtXgkSeqZ7ODJEoqmofoSabQ6dvT0RVcdRdLQ4wgb047gCDvl54772h9v2dGUY2JWiJgVSvscmU7WLLNIVx09l1pe1lKHHkfYnG4+iOXEURWd9ol6i/PHE2w5guXE3cS9AKebD3I2HITmEMX54zndfIhg6xFAYVTeWBR0HGF1e45MyjzUHVmzzCLtO3qEafZ5mNCllpcuD9uJYzlxDN2PaYWJ22EM3Y/tmMStNnef7ZjYTtw93qN53WPiVhuqqqIpKjGz9UK6t+7PkQ1kzTKL9NTRcznKS5eHphroqoFpRfHoAdon6jX0HHdf+8S9umpg2c1oaj6GnoOh5yAiAoHA68lDQe/1HJlO5rPMMskkwcmOnr7mMOxYXhqaHGG7yXkhNVFv+33Jx2dH2Jw+fZKSknL3mGSCYM+FJSN6O0cmkEvhZpmeApqiaZe0js6llpcuD1XRUtbFSfm5w77kNl3zuYFPVTS8euqQst7OkekGrc3SNE1qa2uZMmUKVVVVTJs2jaeffrrX2l4kEuGXv/wlc+fO5frrr2f69OlMnz6dxx9/nJaWlst09UNXshMmfuhAojNG9lpLUr8YtJrlokWL2LlzJ++88w5jxoxh//79zJo1i2AwSG1tbbflfve737Fo0SK2bNnCvHnzAHjvvfe4/fbbeeWVV/jTn/6Ex+O5XLcx5HQ3W0eSpEszKDXLXbt2sWnTJlasWMGYMWMAqKqq4sEHH2TNmjUcP368x/Jz5851AyVATU0N9913H++99x5vvPHGgF77UCc7YSRpYAxKsNyyZQuQCHrt3XHHHZimyQsvvNBt2Xnz5vHKK6902p4Muo2Njf14pcNPcraOMWGKHAcpSf1oUB7D9+zZg2EYjB49OmX7hAkT3P3dUVUVVe0c4w8cOIDH4+Hmm2/u34sdhjp2wnTV4ZPcJhzZppnpMrXn+nIblGAZDAbJy8tDUZSU7fn5+e7+vjh//jz/+Z//yQ9+8AO3hikldDdrJ7lNtIUQo4plDTRDtZ/Nk02zbQbCoARLIUSnQHmxbNvmH/7hH5gxYwarVq3q8dh169axbt0693Vfg/JwJGJRnDNBFJ8P0dyEmnMSwN1mxWIEG06ieH2DfKXSQLDsKGfDQTyaF8tuhmguutb139qyrKz4TlysQQmWxcXF1NXVdQqayaE/6Q6SdhyHRYsWEY1Geemll9D1nm9nyZIlLFmyBEgMPs2GBeWFbRNva0KYJkp+HkZZOYC7rdm2KC4rlzXLDOUIG5pD2I6JpuZTUlDebc2yrxMUss2gBMuamhp27dpFfX19ymPz4cOHAaiuru71HKZp8s1vfhPHcXj55ZcxDNnr25Xu0rO525qbZaDMYKqiUVJwtWyz7AeD0hs+f/58AHbs2JGyffv27Xg8Hu666y53W0NDA47jpBwXiUT48pe/TG5uLlu3bnUD5auvvsrKlSsH+OqHn2R6tvZB0d2myi9PplMVDY/ml4HyEg1KsLzppptYsGABtbW11NfXA/D++++zdu1ali9fzrhx4wDYtm0b5eXlPPTQQ27Z1tZWPve5z3Hs2DFuu+02tmzZwubNm9m8eTOvvPIKR48eHYxbkqSMl405LNsbtBk8GzZsYPXq1cyePRvDMLBtm1WrVqUExsLCQvLy8igrK3O3bdy4kbfffhuABQsWdDrvt7/97YG/eEnKMrJXXWYdGuzLGHSyUV9K6umzYNoRTjUfvJAjM0pJwUQ3G1Em6SkuyOS/kiT1qn2OzGzKYdmeTNEmSVKvZK+6rFlmFbnYmHQxOiYCzsZACbJmmTXkYmPSxZAdO5+SNcssIRcbky5G+8XPsmlxsq7IYJklZJ5L6WLIjp1PycfwLNFx2iOQaL+UKdqkHsiOnU/JYJlFknku27dfyhRtUm+ycXGyrsjH8CyU0n5pWbL9UpLSIINlFkppv9R12X4pSWmQj+FZKKX9UqZok6S0yJpllpIp2iSpb2SwlCRJSoMMlpIkSWmQwVKSJCkNMlhKkiSlQQZLSZKkNMhgKUmSlAYZLCVJktIgg6UkSVIaZLCUJElKgwyWkiRJaZDBUpIkKQ0yWEqSJKVBBktJkqQ0yGApSZKUBhksJUmS0jBowdI0TWpra5kyZQpVVVVMmzaNp59+GiFEWuV37NjBzJkzmTZtGldffTX33HMPZ86cGeCrliQpWw1apvRFixaxc+dO3nnnHcaMGcP+/fuZNWsWwWCQ2traHstu376dO++8k+eff55vfetbRKNRvvzlL3Prrbfyl7/8hZycnMt0F5IkZQ0xCP74xz8KQKxfvz5l+8MPPyw8Ho+oq6vrsfzVV18tZs2albLto48+EoBYvXp1WtcwSLc+5Jw5c2awL0EaIuRnoee4MCg1yy1btgAwd+7clO133HEH//Iv/8ILL7zA97///S7L7t69m4MHD/LNb34zZfukSZOoqKhgy5YtPPzwwwNz4UOcHYlgnjmJ45iABpaJlleAY8chHMbxehFNTagBP1rRSOxwiOjRY0Ti4zFGlKIoYEUjWPXHED4vev5IwEZBQ8srwI6GsM+fRysYgadohFy7Z4iKWiFaIqex0WgN1SNQGJE/Go/qBRwURSdqhjDtGLYdxxY2ioDGc0GaRREKAo/uR1M9+IwCECa65kcIB8uJYTs2Hs2D35OPV8/NmrXEByVY7tmzB8MwGD16dMr2CRMmuPt7Kgswfvz4TvsmTJjAm2++STwexzCya8VCOxKh6Vc/J7r3zzgtjRCPgT8XNA0QYDsQagZFAV2HgpHQ1gzRGGcLi/D9r5vwTrmO1t/8EufcGbAtGHkFem4eakEh6ohRWMeP4LQ2o+aPoOCr38Zfdb0MmENM1Arx5odP0xg+yZnQEQQxQMVDHiPzx6KgErfaiJlthM0mBA4C80JpB05AoitDw1By8eg6ub4SHDuGqvmIxM5f+AgFGFM0nQkl/x9XFk3NioA5KMEyGAySl5eHoigp2/Pz8939PZVtf2zH8rZtc/78eUpLSzvtX7duHevWret0rkxgB09hnm4A2wbTAssCx4ZYFBQVNBXiJni9if0t58B0QNUgGiP6t3qiOYXQciGg2g60tmApOnj9UH8CmkOgenDa2mg8dphQcRmK1zfYty610xQ5QUv4HLZtIvh0PXiLCNFYC5riIWqFsJzYhUBpAU6HswjAwhEmMdPE0CJYVhRVi2PZMRRFBaK0hBoJavXo8SJ0LfM/B4MSLIUQnQJlX8oCF1V+yZIlLFmyxC1fXFx8UdcwFNm5uTSVlBE9VY/j8QAXAmHAAAQ4DhieCzVLDfKTNcs4+L34rhyNd1wlrfkFOOfPJIJrfgF6jh/V70MtGonlmImaZU4RBRUT8JeVy5rlEJNn+TncOJLGSBwF40LNEnT8+L35KKgoKkQtFTMeRUHHITkCJRk0FUBDUzzouo7hCaCrKprmQwgLRVHw6D4KcosoHjWakqJyWbMcKMXFxdTV1XUKmi0tLe7+nsoCNDc3d9rX0tKCpmkUFRX18xUPfZrfT+H/+Q7mnC/0qc3y3NFjjBr/aZulb+q1ss1yGPPpudw29aGLbrPMHyHbLLszKMGypqaGXbt2UV9fz5gxY9zthw8fBqC6urrHsgBHjhzptO/w4cNMmTIFr9fbz1c8PGh+P9q4zm25Kca2+3kktPkK8Lf7z8nw+TEKR3R7fooypzaeqXx6Lr68XADK8iq6PKbAd0WnbbkEM+ppq78NyqD0+fPnA4mB5e1t374dj8fDXXfd5W5raGjAcT5tU6mpqWHixImdyh46dIhjx45x9913D+CVS5KUtS7X+KWOFixYICoqKsSJEyeEEELs379f5OfnixUrVrjHbN26VQDi/vvvTyn7+9//XmiaJjZt2iSEECISiYjPfe5zYvLkySIUCqX1/oN460OKHFsnJcnPQs9xYdCmO27YsIF77rmH2bNnU1VVxfz581m1ahUrV650jyksLCQvL4+ysrKUsnPnzuW1117j2WefZdq0aUyfPp3i4mLeeustOXtHkqQBoVyIpllHUZS056FnsmBQtlNJCfKz0HNckFmHJEmS0iCDpSRJUhpksJQkSUqDDJaSJElpGLR8lkPBxU65lCQp+2RtsJQ94Qk33HAD//M//zPYlyENAfKz0DP5GC5JkpQGGSwlSZLSIINllkumrJMk+VnoWdbO4JEkSeoLWbOUJElKgwyWGWzGjBmUlpbKIVKS1A9ksMxg7777LkuXLh3sy5Auo6amJh5//HH27t07YO+xd+9eHn/8cZqamgbsPYYiGSwlKYM0NTWxcuXKAQ+WK1eulMFSkiRJ6kwGywzyX//1X1x33XWUlpYyY8YMHnvssZQlOZL27t3L5z//ecaNG8eECRO48cYbefXVV939t912G4ZhkJubS3V1NSdOnADg0UcfpaKigqKiIpYtW3bZ7ktKz8aNG7nzzjuBxN+qurqa6upqdwmWN998k1tvvZWrrrqKyspKbr/9dnbt2pVyjtOnT7Nw4UKmTp1KTU0N1dXVPPjgg3zyyScAPPDAAzz66KMA3Hnnne57ZEUt83KkapcG3o4dO4SmaWLZsmXCcRwhhBBbtmwRJSUlKanyd+/eLQKBgFi8eLGwLEsIIcSmTZuEoijit7/9rXvcN77xDZGXl9dpmY777rtPPPPMM5fhjqSLcfToUQGIjRs3pmx/5ZVXhKZpYuXKle62n/zkJ8IwDPGnP/3J3TZnzhwxZ84cEY/HhRBCNDQ0iIkTJ6acb+PGjQIQR48eHchbGXJksMwQM2bMEEVFRSISiaRsv+WWW1KC5W233SZycnJEc3NzynE33nijmDhxovv6D3/4gwDE+vXr3W1tbW2irKxMNDU1DdBdSJeqq2DpOI6orKwUFRUV7n+QQghhWZYYPXq0mDNnjrstJydHLF68OOWcL730knj33Xfd19kaLOVjeAZoa2vjL3/5C9deey0+ny9lX/tlhcPhMDt37uS6664jPz8/5bhrr72WQ4cOcfz4cSDxKD5x4kR+8YtfuMds27aNO+64g4KCggG8G6m/HTp0iE8++YRZs2ahtVvrXdM0pk6dyttvv41pmkBifasNGzYwb948Xn31VSKRCF/60pf4zGc+M1iXP2RkbdahTNLY2IjjOIwY0Xm978LCwpTjbNvmgw8+6LQ2eygUoqSkhGAwyNixicXFFy9ezI9+9CP2799PVVUV69at48knnxzYm5H6XTAYBBJLTXf8uzc3N1NUVERjYyNXXHEFW7du5ZlnnuH555/nC1/4Arm5uSxYsIA1a9Z0+g8228hgmQGKiopQVZVz58512tfY2JhynKZp3Hjjjbz22mu9nnfhwoWsWLGCX/ziF3znO9+hpaWFmTNn9uu1SwMvuQjZV77yFX72s5/1eKxhGCxbtoxly5bx4Ycf8vOf/5y1a9fS2trKpk2bLsflDlnyMTwD5OTkMGPGDPbt20c0Gk3Zt2/fPvfnQCDALbfcwv79+7EsK+W4v/71r9x9990p24uLi/nSl77E5s2beeqpp7j33nsH9kakS+bxeADcURB1dXWcPXuWyspKdu/e3en4t956iwceeMB9PX/+fPfnqVOn8m//9m/8/d//fcrnqON77N69m4MHD/b/zQwxMlhmiNWrV9Pa2sqPf/xjN7Hx1q1bO31BfvrTn9Lc3Mzy5cuxbRuAs2fPsnTpUioqKtD11IeNe++9l6amJjZv3syCBQsuz81IF62kpIRAIOAO91q7di0bNmxg7dq17Nmzh6eeeso99tixY9x3331UVVW527Zt28bmzZvd16dPn+bDDz9k7ty57rbKykoATpw4gRCC+++/n3fffXegb23wDXYPk9R/XnvtNVFdXS2uuOIKUVNTI773ve+Jhx9+WADi2muvFc8//7wQQoj9+/eLL37xi+LKK68U1dXV4oYbbhBPPfWUO+SovWRP6je+8Y3LfTvSRdqwYYOoqKgQVVVV4qabbhJHjhwRQgixc+dO8dnPflaMHj1a1NTUiL/7u78Tv/rVr1LK/vSnPxUzZ84U06dPF9XV1WLatGnisccec4cSJd13331i3LhxYurUqeLrX/+6iMVil+3+BotM0SZJkpQG+RguSZKUBhksJUmS0iCDpSRJUhpksJQkSUqDDJaSJElpkMFSkiQpDTJYSpIkpUEGS0mSpDTIYClJkpQGGSwlSZLS8P8DTsKIfct0EqcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "utils.plot_worker_agreement(\n",
    "    dev, test, os.path.join('img', 'human-agr-round1.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples for the paper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>model_0_label</th>\n",
       "      <th>responses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Good food nasty attitude by hostesses .</td>\n",
       "      <td>negative</td>\n",
       "      <td>mix, mix, mix, neg, neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Not much of a cocktail menu that I saw.</td>\n",
       "      <td>negative</td>\n",
       "      <td>neg, neg, neg, neg, neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>I scheduled the work for 3 weeks later.</td>\n",
       "      <td>negative</td>\n",
       "      <td>neu, neu, neu, neu, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>I was very mistaken, it was much more!</td>\n",
       "      <td>negative</td>\n",
       "      <td>neg, pos, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>It is a gimmick, but when in Rome, I get it.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>mix, mix, mix, neu, neu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Probably a little pricey for lunch.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>mix, neg, neg, neg, neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>But this is strictly just my opinion.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>neu, neu, neu, neu, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>The price was okay, not too pricey.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>mix, neu, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The only downside was service was a little slow.</td>\n",
       "      <td>positive</td>\n",
       "      <td>mix, mix, mix, neg, neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>However there is a 2 hr seating time limit.</td>\n",
       "      <td>positive</td>\n",
       "      <td>mix, neg, neg, neg, neu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>With Alex, I never got that feeling.</td>\n",
       "      <td>positive</td>\n",
       "      <td>neu, neu, neu, neu, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Its ran very well by management.</td>\n",
       "      <td>positive</td>\n",
       "      <td>pos, pos, pos, pos, pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            sentence model_0_label  \\\n",
       "0            Good food nasty attitude by hostesses .      negative   \n",
       "3            Not much of a cocktail menu that I saw.      negative   \n",
       "6            I scheduled the work for 3 weeks later.      negative   \n",
       "9             I was very mistaken, it was much more!      negative   \n",
       "1       It is a gimmick, but when in Rome, I get it.       neutral   \n",
       "4                Probably a little pricey for lunch.       neutral   \n",
       "7              But this is strictly just my opinion.       neutral   \n",
       "10               The price was okay, not too pricey.       neutral   \n",
       "2   The only downside was service was a little slow.      positive   \n",
       "5        However there is a 2 hr seating time limit.      positive   \n",
       "8               With Alex, I never got that feeling.      positive   \n",
       "11                  Its ran very well by management.      positive   \n",
       "\n",
       "                  responses  \n",
       "0   mix, mix, mix, neg, neg  \n",
       "3   neg, neg, neg, neg, neg  \n",
       "6   neu, neu, neu, neu, pos  \n",
       "9   neg, pos, pos, pos, pos  \n",
       "1   mix, mix, mix, neu, neu  \n",
       "4   mix, neg, neg, neg, neg  \n",
       "7   neu, neu, neu, neu, pos  \n",
       "10  mix, neu, pos, pos, pos  \n",
       "2   mix, mix, mix, neg, neg  \n",
       "5   mix, neg, neg, neg, neu  \n",
       "8   neu, neu, neu, neu, pos  \n",
       "11  pos, pos, pos, pos, pos  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.sample_short_examples(train, 'model_0_label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>model_0_label</th>\n",
       "      <th>responses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>We so wanted to have a new steak house restaurant.</td>\n",
       "      <td>positive</td>\n",
       "      <td>neu, neu, neu, neu, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>As a foodie, I can surely taste the difference.</td>\n",
       "      <td>positive</td>\n",
       "      <td>neu, neu, neu, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>There was however some nice dinner table chairs that I liked a lot for $35 a piece and for the quality and style this was a very nice price for them.</td>\n",
       "      <td>positive</td>\n",
       "      <td>pos, pos, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The waitress helped me pick from the traditional menu and I ended up with chilli chicken.</td>\n",
       "      <td>positive</td>\n",
       "      <td>neu, neu, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I have had lashes in the past and have used some of those Living Social deals.</td>\n",
       "      <td>positive</td>\n",
       "      <td>neu, neu, neu, neu, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Lots of trash cans.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>mix, neu, neu, neu, neu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>They were out the next day after my call to do the inspection and same for the treatment.</td>\n",
       "      <td>positive</td>\n",
       "      <td>mix, neu, neu, neu, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>When we walked in no one was there to sit us, I waited for a minute and then decided just to take a seat.</td>\n",
       "      <td>negative</td>\n",
       "      <td>mix, neg, neg, neg, neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Driver was amazing!</td>\n",
       "      <td>positive</td>\n",
       "      <td>pos, pos, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>We tried:\\n\\nChampagne On Deck - Smooth and easy to drink.</td>\n",
       "      <td>positive</td>\n",
       "      <td>neu, neu, pos, pos, pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                sentence  \\\n",
       "0                                                                                                     We so wanted to have a new steak house restaurant.   \n",
       "1                                                                                                        As a foodie, I can surely taste the difference.   \n",
       "2  There was however some nice dinner table chairs that I liked a lot for $35 a piece and for the quality and style this was a very nice price for them.   \n",
       "3                                                              The waitress helped me pick from the traditional menu and I ended up with chilli chicken.   \n",
       "4                                                                         I have had lashes in the past and have used some of those Living Social deals.   \n",
       "5                                                                                                                                    Lots of trash cans.   \n",
       "6                                                              They were out the next day after my call to do the inspection and same for the treatment.   \n",
       "7                                              When we walked in no one was there to sit us, I waited for a minute and then decided just to take a seat.   \n",
       "8                                                                                                                                    Driver was amazing!   \n",
       "9                                                                                             We tried:\\n\\nChampagne On Deck - Smooth and easy to drink.   \n",
       "\n",
       "  model_0_label                responses  \n",
       "0      positive  neu, neu, neu, neu, pos  \n",
       "1      positive  neu, neu, neu, pos, pos  \n",
       "2      positive  pos, pos, pos, pos, pos  \n",
       "3      positive  neu, neu, pos, pos, pos  \n",
       "4      positive  neu, neu, neu, neu, pos  \n",
       "5       neutral  mix, neu, neu, neu, neu  \n",
       "6      positive  mix, neu, neu, neu, pos  \n",
       "7      negative  mix, neg, neg, neg, neg  \n",
       "8      positive  pos, pos, pos, pos, pos  \n",
       "9      positive  neu, neu, pos, pos, pos  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.sample_examples(train, 'model_0_label')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}