7.4. 动量法.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "7.4. 动量法.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/chongzicbo/Dive-into-Deep-Learning-tf.keras/blob/master/7.4.%20%E5%8A%A8%E9%87%8F%E6%B3%95.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OfTvPUb8finD",
        "colab_type": "text"
      },
      "source": [
        "##7.4. 动量法\n",
        "在“梯度下降和随机梯度下降”一节中我们提到，目标函数有关自变量的梯度代表了目标函数在自变量当前位置下降最快的方向。因此，梯度下降也叫作最陡下降（steepest descent）。在每次迭代中，梯度下降根据自变量当前位置，沿着当前位置的梯度更新自变量。然而，如果自变量的迭代方向仅仅取决于自变量当前位置，这可能会带来一些问题。\n",
        "\n",
        "###7.4.1. 梯度下降的问题\n",
        "让我们考虑一个输入和输出分别为二维向量$\\boldsymbol{x} = [x_1, x_2]^\\top$ 和标量的目标函数$f(\\boldsymbol{x})=0.1x_1^2+2x_2^2$。与“梯度下降和随机梯度下降”一节中不同，这里将$x_1^2$系数从 1 减小到了 0.1 。下面实现基于这个目标函数的梯度下降，并演示使用学习率为 0.4 时自变量的迭代轨迹。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b_OULOd4BNzN",
        "colab_type": "code",
        "outputId": "97c7e01b-5bc2-4094-96c6-95b4991f3981",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 64
        }
      },
      "source": [
        "%matplotlib inline\n",
        "import math\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "from IPython import display\n",
        "import matplotlib.pyplot as plt\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import losses\n",
        "from tensorflow.data import Dataset\n",
        "import time"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KZw1gd8VBTP9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.enable_eager_execution()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KgDunhF3BWS1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "eta=0.4\n",
        "def f_2d(x1,x2):\n",
        "  return 0.1*x1**2+2*x2**2\n",
        "\n",
        "def gd_2d(x1,x2,s1,s2):\n",
        "  return (x1-eta*0.2*x1,x2-eta*4*x2,0,0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j6VgDnjlB1A8",
        "colab_type": "code",
        "outputId": "f44440e4-bb89-4fa6-b5e4-dfa72df12c3b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        }
      },
      "source": [
        "def show_trace_2d(f,results):\n",
        "  plt.plot(*zip(*results),'-o',color='#ff7f0e')\n",
        "  x1,x2=np.meshgrid(np.arange(-5.5,1.0,0.1),np.arange(-3.0,1.0,0.1))\n",
        "  plt.contour(x1,x2,f(x1,x2),colors='#1f77b4')\n",
        "  plt.xlabel('x1')\n",
        "  plt.ylabel('x2')\n",
        "def train_2d(trainer):\n",
        "  x1,x2,s1,s2=-5,-2,0,0 #s1,s2是自变量状态,本章后续几节会使用\n",
        "  results=[(x1,x2)]\n",
        "  for i in range(20):\n",
        "    x1,x2,s1,s2=trainer(x1,x2,s1,s2)\n",
        "    results.append((x1,x2))\n",
        "  print('epoch %d,x1 %f,x2 %f'%(i+1,x1,x2))\n",
        "  return results\n",
        "\n",
        "show_trace_2d(f_2d,train_2d(gd_2d))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "epoch 20,x1 -0.943467,x2 -0.000073\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEGCAYAAAB7DNKzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOy9eZwc1X2u/5yq3ns2zSqNdrSBwBIC\ngTGbTcCGEGOMF37GN45xnBDHzvWNHceJ41yHy3V+3rJ4TWLsON4diB0MRhgveAEbZCNACAECCWmk\nkTTSbD1b78u5f5yq7ppRzz49PRp9n8+nOVWnaqrPIKneOku9r9JaIwiCIAjjYVW7AYIgCMLCRoRC\nEARBmBARCkEQBGFCRCgEQRCECRGhEARBECbEV+0GVILm5ma9Zs2aajdDEAThtOGJJ57o1Vq3lDu2\nKIVizZo17Nq1q9rNEARBOG1QSh0e75gMPQmCIAgTIkIhCIIgTIgIhSAIgjAhIhSCIAjChIhQCIIg\nCBMiQiEIgiBMiAjFQmHP3fDP58HtDabcc3e1WyQIggAs0vcoTjv23A0/eC9kk2Z/sNPsA2y5uXrt\nEgRBQHoUC4OH7iiJhEs2aeoFQRCqjAjFQmDw6PTqBUEQ5hERioVA/Yrp1QuCIMwjIhQLgas/ApZ/\ndJ0/bOoFQRCqjAjFQmDLzbDyktJ+TRvc8FmZyBYEYUEgQrFQsCyItprtqz4sIiEIwoJBhGKhEOuA\ntVdCqB6OP1Xt1giCIBSpqlAopb6ilOpWSu0d57hSSn1WKXVAKbVHKXXBfLdxXshnzQqnxrXQvk2E\nQhCEBUW1exRfBa6b4PjvAhucz23Av85Dm+afwaOg87BkjRGKk89CLl3tVgmCIABVFgqt9cNA/wSn\n3Ah8XRt2Ag1KqWXz07p5JNZhSlcoClk4WbaTJQiCMO9Uu0cxGcuBTs/+UaducTFWKECGnwRBWDAs\nGq8npdRtmOEpVq1aVeXWTJNYB9gBqF0GyoJIkwiFIAgLhoXeozgGrPTsr3DqTkFrfafWervWentL\nS8u8NG7OiHVAwyqwbFDKmdDeXe1WCYIgAAtfKO4D/sBZ/XQJMKi17qp2o+acWIcZdnJp3wbdz0Mm\nUa0WCYIgFKn28tjvAI8Bm5RSR5VS71RKvUsp9S7nlAeAg8AB4EvAu6vU1MpSTih0/rSY0M4XNJ39\nImiCsJip6hyF1vqWSY5r4D3z1JzqkIxBauBUoQAzT7Hy4qo0qxxaa44NJNndOcCeo4M83TnA3mOD\nFDQ8c/tr8NkLvYMqCMJMWDST2actscOm9ApF7TLj91TlCe2hVJanOwd4unOA3Z0D7O4cpHfEvN8R\nsC3Oaa/jjReuYMuKBvJay18mQVikyL/tauNdGuuiFLRfMK9CkS9oDnSP8NSRGE8eifHUkQEO9Iyg\ntTl+VkuUKzc2s21lA1tXNnD20joCPulBCMKZgAhFtXGFomH16Pr2bfDig5AegWDNnH/tYDLL7s4B\nnjgc4ylHGEbSOdOUiJ9tKxu4YWs75zvCUB/2T3JFQRAWKyIU1SbWYd6bCNWNrm/fBmg4sQdWXzrr\nrzkaS7DzYD9PHO7nicMx9neb3oKlYNPSOl6/rZ1tK5dwweolrGmKoJSa9XcKgrA4EKGoNmNXPLm0\nn2/K40/NSChi8QyPvtTHrw708uhLvRzuMyuT6kI+Lli9hBu2tHPh6iVsWdlATVD+GgiCMD5yh6g2\nsY7SKicvNa1Qt2LK8xSpbJ4nDsd4ZH8vvz7Qy97jg2gNNUEfl5zVyNtfsYZL1zexsbUWy5LegiAI\nU0eEopoU8jDYCefeVP54+/njCoXWmhdPjvDI/h4e3t/Lbw72kc4V8FmKC1Yt4c+v3sjlG5rYuqJB\nlq0KgjArRCiqydAxKOTKDz2B6Wnsux9SgxCqJxbP8MiBXh5+sYdH9vdwcsgsVV3XEuWWi1dxxYZm\nLjmriagMJQmCMIfIHaWalFsa60G3b0MB//WDHXyzezV7jg6gNdSH/Vy+vpkrNzZz+YYWljeE56vF\ngiCcgYhQVJMyQqG15umjgzzwTBe/2jPMA8CBpx/Gan8b/+vqDVy5sYWtKxqwZZ5BEIR5QoSimsQ6\nwPKh69rZfSTG/Xu6eHDvCY4NJPHbisvXtzPSs4K/WBkncMtl1W6tIAhnKCIUVUJrzcCx/Wj/Um74\n1CMcG0gSsC2u2NDM+1+9kWvOaaM+4oe7t1fdykMQhDMbEYp5RGvN813D3L/nOPfv6eKzI88ywhI2\nnVXL+1+9kVef20ZdaMwb0O3b4LnvQ6IfIo3VabggCGc0IhTzwLGBJN/ddZT7nj7GSz1xbEtx2fpm\nzs71o865nMtvumj8H/Y6ya6/en4aLAiC4EGEokIUCppHDvTyjccO87N9J9HAxWsaecdla/nd85bS\n5EvDx2PQctbEF1q21ZQiFIIgVAkRijlmIJHhu08c5Zs7D9PRl6ApGuBdr1zHLRevYmVjpHTiiRdN\nOd47FC7hBmhcJ/MUgiBUjaoKhVLqOuAzgA18WWv98THHbwU+RSkn+/Na6y/PayOnyN5jg3z9sQ7u\n3X2cdK7AhauX8L5Xb+S685YS9Nmn/sAk71CMon0bHNk5h60VBEGYOlUTCqWUDXwBeDVwFHhcKXWf\n1vq5MafepbX+s3lv4BTI5gs8uPcEX3u0g12HY4T9Nm+4YDm/f8lqzm2vn/iHpysUe78LI93GA0oQ\nBGEeqWaP4mLggNb6IIBS6j+BG4GxQrHgGE5l+dqjHXz9scN0D6dZ1Rjhb3/vHN584UqzpHUqxDog\n1GCGliajOKG9Gza+ZsbtFgRBmAnVFIrlQKdn/yjw8jLnvVEpdSXwIvA+rXVnmXNQSt0G3AawatWq\nOW6qIZXN882dh/nCzw8QS2S5cmMLH3/jal61sXX6jqzj2YuXY9lWQJl5ChEKQRDmmYU+mf0D4Dta\n67RS6k+ArwG/U+5ErfWdwJ0A27dv13PZiFy+wPeePMqnf7qfrsEUV2xo5i+v3cSWFVPoDYxHrAPa\nzpvaucEaaNk0/xPae+6Gh+6AwaNQvwKu/ghsuXl+2yAIQtWpplAcA1Z69ldQmrQGQGvd59n9MvDJ\neWiX9/v50bMn+OSPXuBgT5zzVzbwjzdv5dJ1zbO7cCEPA0fg7NdO/Wfat8FLP5/d906HPXfDD94L\n2aTZH+w0+yBiIQhnGNUMKngc2KCUWquUCgBvAe7znqCUWubZfR3w/Hw17jcH+7jpXx7lXd98Eksp\nvvi2C7nn3ZfOXiQAhrsgn5n60BMYoRg5AUNds//+qfDQHSWRcMkmTb0gCGcUVetRaK1zSqk/A36E\nWR77Fa31s0qpO4BdWuv7gPcqpV4H5IB+4NZKt6uzP8Ht9z3LQ/u6WVoX4pNv3MIbLlg+t+E/01nx\n5OJ9Q7tu2cTnzgWDR6dXLwjCoqWqcxRa6weAB8bUfcSz/SHgQ/PUFr735DFuv+9ZAP7qurN5x2Vr\nCPnLvAMxW2YiFG3ngbKNUJx9/dy3aSz1K8xwU7l6QRDOKBb6ZPa8MJDI8Df3PMMDz5zg5Wsb+ceb\nt7JiSWTyH5wpsQ5z05/OTTcQgdZz5m9C+7I/hwf+YnSdP2wmtAVBOKM444Xi0QO9vP/up+mLp/mr\n687mtivPqnwoUOywEQl7iu9cuLSfDy88CFqDqnAb3bbVtMHISbN9wdtlIlsQzkDOWKFI5/L8w49e\n4EuPHOKslihffvtlnLd8krep54pYByxZPf2fa98GT33TzBM0rJz8/Nmwbwc0rIb/9TToAnx+u7ER\nmQ+REgRhQVHNVU9V4/hAkpu+8ChfeuQQv3/JKnb8zyvmTyRgei/beSlOaD85l605lfQwHPwFnP17\nRhQsGy5/H3Tthpcequx3C4Kw4DjjhKJrMMktX9pJZ3+Cf3/7dj76+pcRDlRgwno8MnGId89MKNrO\nA8tf+XmKAw9BPm2EwmXLW6BuOTzyT5X9bkEQFhxnlFCcGExxy5076R/J8PV3XszV57TNfyNih005\nE6HwBaFtc+WF4oUHINwIKy/xfHcALn0vHP41HH6sst8vCMKC4owRipNDKW750k56RzJ87Z0Xs23V\nkuo0ZCZLY720bzNCoefUpaREPgsvPgibfhfsMVNYF/wBRJrhkX+szHcLgrAgOSOEonvI9CS6h1J8\n7Q8v4oJqiQR4hGLtzH6+fRukBiF2aM6aNIrDvzbX9w47uQQi8Ip3w4GfQNfTlfl+QRAWHIteKIZT\nWW750k5ODKX42h9ezIWrG6vboFgHBOsgPEOx8r6hXQn27QBfGM66qvzxi/7ItL/SvYo9d8M/nwe3\nN5hyz92V/T5BEMZl0QvF5392gJd64nz57dvZvqbKIgGlpbEzXWLacg7YwcoIhdZGKNb9juk9lCNU\nDxf/MTx3H/S8OPdtgJIh4WAnoEuGhCIWglAVFrVQHOwZ4Su/PsSbL1wxN2Z+c8FMl8a6+AKw9DwT\nYjTXdO2GoWPlh528XPJu8IXgV/88920AMSQUhAXGohaK/3v/c4R8Nh+87uxqN8VQKMDA4dkJBTgT\n2rvN9eaSfQ+AsmDjdROfF22GC2+FPXeVVnHNJWJIKAgLikUrFD/bd5Kfv9DDe6/eQEttsNrNMYyc\nhFxqDoTiAsgMQ/9Lc9KsIvt2wKpLIdo0+bmX/k8jKo9+bm7bAOO744ohoSBUhUUrFB+9/3nOaony\n9kvXVLspJWa7NNalEhPa/Qeh+9nJh51c6pfD+bfAk1+H4ZNz1w6tzTsc5Wg8y4Q+zSUyaS4Ik7Io\nhSKTK3CwN847L19LwLeAfsXZLo11ad4I/sjcCsU+x+19Ohbml/05FLKw8wtz146d/wIn98KWW6B+\nJaBMufE6OPRLuOttkEnMzXfJpLkgTIlFaQqYzJqnzq2zybSuBLEOije+2WD7YOmWORaKHcYiZDq9\nnaZ1cO4b4PF/N6IRmeWqsuNPwU/+Djb9Htz0r6euDPvNnfDDD8LXXwe33DW1IbKJmGjSfC5cciVz\nXFgkVPVxWyl1nVLqBaXUAaXUX5c5HlRK3eUc/41Sas1UrpvM5vHbig1tNXPd5NkR6zA3DF9g9tdq\n32ZeepuLoZh4L3TunPqwk5cr3g+ZEfjtl2bXhtQQ/Nc7jK35jZ8vv3z45bfBzV+Hrj3wldeUemgz\nZdxJ887ZX3s6vZWpDH9Nds5sjo93bC7qp1J3//tl+G+Bo3SlrCAm+2KlbOBF4NXAUUyG9i1a6+c8\n57wb2KK1fpdS6i3ATVrr/2+yazetOUdf8v472fHeKyrU+hny79eanIdb75/9tZ6+C+65Dd690wQa\nzYanvgn3vgf+5GFYtnX6P/+dW+DIY/DneyE4A3HWGv77j2Hv9+DWHbD60onPP/wYfOctYAfgf/yX\nyemYDpkEPPpZ+MXHgQn+/q+5As5/K2y+EQLRqV8/3gtfuBgSfaceizbDH//CPDAoVRIUb8/GH4Yb\nPlvqfTz9n/CD/2UWQrj4wvA655zJrjHRcSh/bOtb4elvz67e8pvfMZ+ZuG4sY39/YV5QSj2htd5e\n9lgVheIVwO1a62ud/Q8BaK0/5jnnR845jymlfMAJoEVP0ujo8o363Z/+Lz715hnc9CrJP2yCDdfA\njXMwpt/zInzhInj9v5qb2Wz4zi1w4hn482dm9iJg5+Pw79fAaz5qVkNNl6e+Bfe+G676MLzyg1P7\nmZ4X4JtvhGTM9DLWXz35z2gNz/43/PgjMHQUlm838yHeG7A/DFffblaV7f62meQP1MC5r4fzf9/0\nDMYOJ531Kuj4lbE/6fg19Dw/eVv8UWjZaH6PbJk5F1/YGEAOdcHw8fGvE22FZD8Ucqces/zQsMrY\nvegyS6mVZf6853qBwFxQvxLet7farTijmEgoqjn0tBzwhjIfderKnqO1zgGDQNmBaaXUbUqpXUqp\nXbpQKM5TLBgyCRg5MfsVTy5N680NbLbzFJk4vPQz2HT9zN8WX3kRrL0SHv08ZFOTn++ldz888AHz\n9H7FX0x+vkvLJnjnT8z/z2/fDLu/M/H5XU/Df1wP3/1DY59y6wPwxw/B6z43etL8hs/CJe+CK/8S\n/ueT8I4fwubXw9574D+ug/++bfRw0n/fBv+wAb77DvPkX9duxCPaWr4d0Vb4vX+CC94GoYbyIgGQ\nSxqrlHXjWKm4nH19eZEAs9Cg/fzyIgGmfiGKBMg7MwuMRTOZrbW+E7gToGXtZv3s8aEqt2gMA0dM\nOdsVTy6WBcvOn71QvPQz80Q9k/kJL1d8wEwy7/4WXPTOqf1MNmXmJXwheMOdJiBpOtQtg3c8YFZC\nff9d5sm7fuXoJ/7L3wcn9sATXzMC8dp/NpGu7ndtuXn8IQ6lzDDY6kvhdz8Bn36ZeXofhTa2Jr9/\njxm2cx1361eWH9K59u9Hf98/n+cIzxjqV8IffN9sH3p4/HNu+IzJDxnv+Ju+Ap2/Hf84lD+mbNBl\nRGS69TNF3plZUFSzR3EM8C7/WeHUlT3HGXqqB8oM/I4mHLA51BtnOJWdo6bOAQOzyKEYj/bzzZBR\nfha/574d5sl2snmByVh7pRnK+fWnIT/OE+5YfvK/4eQzcNO/mSfxmRCqh//xXXjZzUYgvv+u0U/8\nO95vROKSP4X3Pgnb/3D6ggRm7iUZK38sNQQrLhxty77lZtM7GdtbGStKV3/ECIgXf9jUT/Wc2Rwf\n79iFt86+3vKbeaTJ6sYy9vcXqk41heJxYINSaq1SKgC8BbhvzDn3AW93tt8E/Gyy+QmAsN/cCJ5b\nSL2KuXrZzkv7NtMb6J7CmHg58jl44YfmHQXbP7u2KAVXfsD0nPZ+b/Lz9+2A394Jl7wHNl47u+/2\nBeCmL0KwtvxQSk0bXPexmTv2uoz3lDte/ZabzTj77QOmLNdzmYqgTHbObI6Pd+y1/zT7+tf/i5mP\nm6xu+zsnF1ShqlRtMhtAKXU98GnABr6itf57pdQdwC6t9X1KqRDwDWAb0A+8RWt9cLLrnn/BhXrg\nNXfwl9du4j1Xra/krzB1HvyQebL9m2MznwsYS99L8LkLzD+sC98++fljOfQwfO0GuPkbsPl1s29P\noQD/drkZgvjTx8zwWDkGj8K/XmZcdN/5E5PcNxfc3kD5VUzK3Kxny1RWKAnCacpCncxGa/2A1nqj\n1nqd1vrvnbqPaK3vc7ZTWus3a63Xa60vnopIAPgsxWXrm7jz4YPE4hMsw5tPXNfYuRIJMJYWwfqZ\nz1Pse8DMD0xlxdBUsCzzXkXPPnhhR/lz8jn43h+ZCdg3/cfciQRM/4l/ukx1OEkQFhkLyN9ibvnI\na89lJJ3jH3/yQrWbYpitvXg5lDLzFDMRCjd74qxXTe8dgck49yYjYA//Q/m41oc/ad65eO0/mze7\n55KpjPfPlqkMJwnCImPRCsWmpbW87ZLVfPs3R6o/V6F1ZYQCzDzFyWchl57ez514BgaPzH6101gs\n29h5dO02K6q8HHoYfvlJ83JWJW6w8sQvCBVh0QoFwPuu2Uh92M/tP3iWas7FEO8x6+UrJRSFrBGL\n6bBvB6Bg4+/OfZu2vgVq2+GRfyrVxfvMOwdN6+D6T839d7rIE78gzDmLWijqI34+cO0mfnuon3//\n1aHqNaQSK55cZmo5vm8HrLoEalrmvk2+IFz2Xjj8Kziy0/Sovv+nxtLiTf8xM5sPQRCqxqIWCoC3\nXLSKa89t46M7nufrj3VUpxGVFIqGVSa/YTpCEesw7y/M9bCTlwv+wNhUfP1G+D8NsP9Hxml22ZbK\nfacgCBVh0QuFbSk+d8sFvHpzGx+591m+8VjH/DfCFYqGVXN/baVg+QXTy9B+4Yem3DSN7Inpsm+H\nMX7z+ig9f684gwrCaciiFwqAgM/iC2+9gGvOaeV/3/ss39xZgZzniYh1mDF7f6gy12/fBt3PnZqt\nMB77dkDLOXO/6sjLQ3eYuRMvbtaDIAinFWeEUIAjFv/jAq4+u5W//f5evv2bI/P35ZVa8eTSvs28\n5HZiCm6biX7jclrJYSeYIOtBzN4E4XTjjBEKgKDP5l9+/wKu2tTC39zzDB/74fNkcuM4a84l8yEU\nMLV5ihcfNK6hlRaKSr/8JgjCvHFGCQUYsfi3t13ILRev4ou/PMhN//JrDnQPV+4LsykYOl5Zoahd\nZvyMpiIU+3aYYTBXXCrFfLz8JgjCvHDGCQUYsfjYG17GF992IccHkrz2c7/iG491VOZdC9fJtJJC\noZS58U8mFJmEsaQ+exbZE1NFXn4ThEXDosmjmAnXnruUbSsb+MB39/C/732Wn7/QwyfeuIWW2jn0\nH6rk0lgv7dtg/48hPTL+ewoHf2ECcSo97OQyUdaDIAinDWdkj8JLa12Ir956EX93w2Z+daCX6z79\nMD957uTcfcF8CoUuGGuO8di3w5gIrr68sm0RBGFRccYLBYBlKd5x2Vp+8GeX01oX4o+/vot3fvVx\nXjgxB3MXsQ6Tf1wzTjTmXLHsfFMef7L88XwOXngANr7G5DcIgiBMEREKD5uW1nLvey7jr647m992\n9HPdZx7mL+5+mmMDU3w/oRyVsBcvR20b1C0ff56i8zcmxnO+hp0EQVg0VEUolFKNSqmfKKX2O2XZ\n6DGlVF4ptdv5jE2/qwgBn8Wfvmodj3zwKv74irP4wZ7jXPUPv+D/3v8cfSPTdGiFyi+N9TLRhPa+\nHSaCcv0189MWQRAWDdXqUfw18JDWegPwkLNfjqTW+nznMwcRbFOnIRLgb64/h59/4FXcuLWd//j1\nIa785M/59E9fZCQ9xUzoStqLl6P9fOg7AKnBU9ux735Y+0oTFyoIgjANqiUUNwJfc7a/Bry+Su2Y\nlOUNYT715q38+H1XcuXGFj790/1c+cmf8+VHDpLKlsln9pLoh8zI/PYoALqeHl3f/RwMHJZhJ0EQ\nZkS1hKJNa93lbJ8A2sY5L6SU2qWU2qmUqqqYrG+t5V9//0Lufc9lbF5Wx0d3PM8rPvYQH//hPo7G\nEuV/aL5WPLksG+cNbTd7opImgIIgLFoq9h6FUuqnwNIyhz7s3dFaa6XUeG+6rdZaH1NKnQX8TCn1\njNb6pXG+7zbgNoBVqyrg0uqwdWUD3/yjl7PzYB9f/XUHdz78Enc+/BLXnNPGrZeu4RXrmlDuxHXM\nycBYsrpi7RlFtMk41J4iFPfDiovMhLcgCMI0qZhQaK3HnTVVSp1USi3TWncppZYB3eNc45hTHlRK\n/QLYBpQVCq31ncCdANu3b694nN0lZzVxyVlNHBtI8q2dh/nPxzv58XMnWd9aw9suWc0bLlhObdFe\nfJ6EAk6d0B7oNENR1/yf+WuDIAiLimoNPd0HvN3Zfjtw79gTlFJLlFJBZ7sZuAx4bt5aOEWWN4T5\n4HVn8+hf/w6fetMWogGbv7vvWV7+/z/Ert1PkYu0QiAyfw1q32aGvBL9Zv+FB0wp8xOCIMyQagnF\nx4FXK6X2A9c4+yiltiulvuyccw6wSyn1NPBz4ONa6wUnFC4hv82bt6/k3j+7nHvfcxnXv2wZud5D\n7B5p4M3/9ijff+oYycwkk99zQXFC2wky2nc/NG+E5g2V/25BEBYlVfF60lr3AVeXqd8F/JGz/Sjw\nsnlu2pywdWUDW1c2kO8c4mD4ZXQPp/nzu3YTCdhcc04br9vazhUbmwn67Ln/8uIb2k8Z0ej4tcmv\nFgRBmCFntClgRcllsIePsWHbW/n5K1/FzkN9/ODpLn64t4v7nj5OXcjHa85dymu3LOOy9c347Tnq\n3IUboHGdEYoXf2wCjc5+7dxcWxCEMxIRikox2GlM+paswbIUl65r5tJ1zdxx47n86kAvP9h9nB/t\nPcF3nzhKQ8TPtZuX8ntblnHpuiZ8sxWN9m3GsgMFNUuh/YI5+ZUEQTgzEaGoFOO8Q+G3La7a1MpV\nm1pJ5/I88mIvO57pYsczXdy1q5MlET+v2byU6x3RmFlPQxmhGuyEQBT2flfsvgVBmDEiFJViCi/b\nBX0212xu45rNbaSyeX75Yg8PeESjPuznNZvbuPbcpbxiXRPR4BT+uPbcDc97bLEycfiBM0chYiEI\nwgwQoagUsQ6wg2boZwqE/DbXnruUa89dSiqb55H9vfzwmS4e3HuC/3riKH5bcdGaRl65sYVXbmph\nU1tt6cU+Lw/dAfkx5oXZpKkXoRAEYQaIUFSKWId5I9ua/tBRyG/z6s1tvHpzG+lcnl0dMX75Yg+/\nfKGHj/1wHx/74T5aa4NcsaGFKzc2c9n6ZpprnFS+waPlLzpevSAIwiRMKBRKqTqgZaxthlJqi9Z6\nT0VbdrozR66xQZ/NZeuNGPzN9efQNZjkkRd7eXh/Dw/tO8n3njQCsHlZHVdsbOZ9kWWEEsdPvVD9\nilm3RRCEM5NxhUIpdTPwaaBbKeUHbtVaP+4c/iogS2nGw7UXX3XJnF96WX2Ymy9ayc0XrSRf0Dx7\nfJBH9vfy8Is9fOVXh+jSr+fj/i8TUZlSc/xh1NUfmfO2CIJwZjBRj+JvgAsdP6aLgW8opT6ktb4H\nqHBc22lOMgbpoYq7xtqWYsuKBrasaOA9V60nns7x20PbefA3rVzW8QVaCr0c1018IftWBvas4xXx\nDravbmTT0lpsS/4IBUGYGhMJhe1agWutf6uUugq4Xym1Eqi46d5pzXzbiztEgz6uOrsVzn4f8D66\nh1PsOtBH7kAve17q44d7TwBQG/Rx/qoGLli1hAtXL+H8VQ3Uhfzz2lZBEE4fJhKKYaXUOnd+wulZ\nvAr4PnDufDTutKVKQjGW1toQr9+2nNdvW47WmqOxJLsO97OrI8YTh2N89mf70drEeW9sreWC1Q1s\nW7mEbasaWNdSgyW9DkEQmFgo/hSwlFKbXTM+rfWwUuo64C3z0rrTlWrYi0+CUoqVjRFWNka4aZuZ\n2B5OZXm6c5Anjxjh2LGni+/8thMo9Tq2Ob5V569soMldWSUIwhnFuEKhtX4aQCm1Vyn1DeCTQMgp\ntwPfmJcWno7EOiDaAsGaardkQmpDfi7f0MzlG5oBKBQ0h/riPHVkgKeOxHjqyACf//kBCs5A48rG\nMFtXGNHYurKBc9vriARkhbUgLHam8q/85cAngEeBWuBbmGwIYTwGDld92GkmWJZiXUsN61pqeNOF\npteRyOTYe2yI3Z0xdncO8M/LOIQAACAASURBVNSRAe7fY1JsLQUb22rZsqKel61oYPOyWjYtraNm\nKm+QC4Jw2jCVf9FZIAmEMT2KQ1rrQkVbdboT6zDRo4uASMDHxWsbuXhtY7GuezjFns5B9hwd4Omj\ng/z4uZPcvav0Qt+qxghnL63lnGV1nL20lk1La1ndFJWVVoJwmjIVoXgck0B3EdAM/JtS6o1a6zdX\ntGWnK/mciR992eL939NaG+KazSGu2WwyuLXWHB9Msa9riOe7hnj+xDDPdw3x0+dPFoetQn6LjW21\nbGozwrFpqdluqQ2WtyIRBGHBMBWheKcTKATQBdyolHrbbL5UKfVm4HZMit3FnuuPPe864DOADXxZ\na/3x2XzvvDB01GRAnIZDTzNFKcXyhjDLG8JcfU5bsT6ZybO/e5h9J4Z5wfn8/IUe/uuJUu+jPuxn\nU1stG5fWsKG1lg1tpmyuCYiACMICYVKhKHcT11rPdiJ7L/AG4IvjnaCUsoEvAK8GjgKPK6XuW8hx\nqMCCXPFULcIBu/hCoJe+kTQvnhzhxZPDvHBymP0nh7lv93GGUrniOUsifja01rK+rYb1LTWsb61h\nQ1sNS+tCIiCCMM9UKwr1eWCyf/AXAwe01gedc/8TuBE4PYTiDOpRTJemmiCvqAnyinVNxTqtNd3D\nafY7ArK/e4T9J4fZsaeLwWS2eF5N0Me6lijrWmpY2xxldXOUNU0RVjdFqQ/LS4OCUAkW8vKU5UCn\nZ/8oZgVWWZRStwG3AaxataqyLZuIWAdYfqhrr14bTkOUUrTVhWirCxWX64IRkN6RDAe6RzjQM8KB\nk8Mc7I3z2ME+/vupY6Ou0RDxs7oxwqqmqFNGWNVoPkvrQvICoSDMkIoJhVLqp0C5MIYPa63vnevv\n01rfCdwJsH379upZjMQ6oGEVWHbVmrCYUErRUhukpXZ0DwTMHMiR/gQdfXEO98U53JfgSH+CpzsH\neOCZLvKF0l+DgG2xYknYeekwzMolEVYsKW03RPwypCUI41AxodBaXzPLSxwDVnr2Vzh1C5s5shcX\nJiccsIsrqMaSzRc4PpDkSH+Czn63dITk6AADieyo86MBm+VLwqxYEmHFEjMxv9wtG8I01wSlRyKc\nsSzkoafHgQ1KqbUYgXgL8NbqNmkKxDqgXRzYq43ftljdFGV1U7Ts8eFUls7+JJ0xIyDHBpIcjSU5\nFkuyq6N/1MQ6mB7JsoYQ7fVhljWEWN4QZpmz7dbVBn3SKxEWJVURCqXUTcDngBZgh1Jqt9b6WqVU\nO2YZ7PVa65xS6s+AH2GWx35Fa/1sNdo7ZZIDxmJcehQLntqQn83tfja315U9PpTKcnwgyfEBIx7H\nBlIcc/Z3vtTHiaEUhTEDnNGAzdL6EMvqwyytD9FaG6S1NkhLbYjWOnc7KLYnwmlHtVY93QPcU6b+\nOHC9Z/8B4IF5bNrsGDhsShGK0566kJ+6pX7OXlpeSHL5At3DaboGjYicGEzSNZjixGCKrsEUv9rf\nS+9ImtxYNcEISkttkOaaYLFsrgnSVBNwtgM0OfvSSxEWAvJoM5fI0tgzBp9t0d4Qpr0hzIXjvDJT\nKGhiiQzdw2l6htPFsnfElD3DaQ50j/DYwb5T5kxcArZFYzRAU03AlNEAS8aWEbPdEPGzJBLAb08/\np10QJkKEYi4pCoW8bCcYk0XTMwhyzrKJz83mC/THM/QMp+mLZ+gdTtMfz9AbT9M/kqEvnqFvJE1H\nX5xYPMtIOjfutWqDPhqifhrCRjwaIgGWRPw0hP3Uhf3Uh02dKf3UhUxdyG9J70UoiwjFXBLrgHAj\nhOqr3RLhNMNvW8X3SKZCKptnIJGlL54mFs8SS2QYSGSIJbL0x832QDLLQCJLZ3+CgWSWwWQWPcHC\ncb+tzJBb2E9dyEdtyE9tyOd8zHZNsLRfE/QRdfajQR81AR/RoI1PejSLDhGKuUSWxgrzRMhvs7Te\nTJ5PlUJBM5zKMeiIxkAyw2Ayy1Ayx1Aq62ybcjiVYziV5eRQiqGU2U9k8lP6nqDPoiboIxK0iQZ8\nRAI20aCPsN8mHLAJ+21Co7YtQn6bkM8m6G77bYI+y/mY+qDPIuCzCNo2AWdbHIknRmtNrqBJ5wpk\nPJ90Lk/aLbMF0rmJDcEXpVAcG0hy1+NH2LKigQ2tNfP3hBPrgGXnz893CcI0sSxFfcRPfWRmVie5\nfIF4Os9w2gx9jaRyDKdzxJ3PSDpf3I5nciTSeUbSRmCGUzl6htOksnkSmTzJbJ5kJl92sn862JYi\nYFv4bWXEw7bw2RY+W+G3LPw+hc8yx21L4beNuPgsU29bqvixlKm3LLCUcj7m/5vCbCtljqFAoXBH\n6sbKlfe30ho02pRaU9CmrqC152OEPF/Q5LWmUDA3+ILW5PKmPlvQ5AsFss5+Ll8gkzdlrqDJ5gvO\nRxtBcPYn6kVOlUUpFAOJLH/1vWcAY2+9eVkdW1Y08LLl9WxZUc9ZLTVz/yRSyMPAEdj8+rm9riAs\nEHy2RX3EmrHQlCOXL5DKFUhl856PecJNZ0tPvams8yScH/1UPPbGmMkVyOULZJ0baTZvbqDuzTae\ny5ErmJtvrlAgXzA36VyhQKHglJ4ber5gbtoaz00eU7pqoD2y4GbQgxESF+URGYVxHHBFyFaquF8U\nLNvUuyLmsxW2ZeF39kN+C1/Qh98RSZ9tjvlt09Py20YkA7YRz6DflAFfqTcW8lmlnpvfZtsnJviz\nn7M/8QXEue113P0Xr+SZY4M87QTs3PV4J199tAOASMDm3PY6zltez8ucz6zFY+gYFHIy9CQI08Bn\nW9TYlqQiLnAW7Z/OWS01nNVSw43nLwfMk8FLPSM8c3SQZ44Z8fjOb4/wH1kzNhf222xur+Nly+vZ\n3F7Hee31bGirmfpSQ1kaKwjCImXRCsVYbEuxsa2WjW21vNHJg87lCxzsjRfF49njg9y9q7M4aRfw\nWZy9tNaJ86wrbjfVBE/9gpi8bCcIwuLkjBGKcvhs6xTxyBc0HX1x9h4b5NnjQ+w9NshDz3ePyoRu\nrgkWs6BNOlstm3sPErB8ULe8Wr+OIAhCRTijhaIctqVY11LDOs+wFUDPcJoXTgyz78SQifU8Ocy3\nfnOYlDN09Rn/Ti70NXP7N59ig5MNvaHNXCfkF8txQRBOX0QopoibieAN1ckXNJ39CV44OczWB4eI\n5VdwuC/BL17oKS77sxSsaYoWs6Dd8qyWqAiIIAinBSIUs8C2FGuao6xpjsKOEyw95wZ+csMryeQK\ndPTFeeGEyYN+8eQI+7uH+enz3cUwHUvBqsYI61trWN9aazKhW2tY11ojK0AEQVhQyB1pLkgPQ6K3\n6PEU8JXmPkadlsvT0ZsoZkK/1G0E5Jcv9pDNl9ZiL60Lsa41ytrmKGuba1jbHGFNU5QVSyIEfGKP\nIAjC/CJCMRdMccVT0Fc+kS2XL3C4P2FyoR0BOdgb577dx0cF6NiWYnlDmDXNUdY2RUxvpinK6qYI\nKxsj4hoqCEJFEKGYC2b5DoXPtooT6NeeW6rXWtMfz9DRF6ej12RDH+qN09EX58nDsVEOoq6IrG6K\nsKox4pRR1jSbfQnLEQRhplQr4e7NwO3AOcDFWutd45zXAQwDeSCntd4+X22cFhV62U6pkk31hasb\nRx3TWtMXz3DYIyIdfQmO9MW5f08Xg8nR+QbNNcGiiBQ/zn6L5EELgjAB1XrM3Au8AfjiFM69Smvd\nW+H2zI5Yh7EWDy+Zt69UShWT0caKCMBgIsvhfiMenf0JDvfFOdKf4DcH+/j+7mOjjMICPouVS8Ks\nbIywckmElY1hVi6JsMLZrg/7JadAEM5gqhWF+jyweG4+C9BevD7iZ0ukgS0rGk45ls7lORZLcqQ/\nQWcsSWd/giN9CTpjCZ48HBs1LwImCGf5krCT6GYyodsbQrTXm7q2upBMsgvCImahD1xr4MdKKQ18\nUWt953gnKqVuA24DWLVq1Tw1zyHWAW2b5/c7Z0HQZxe9sMoxmDRhN0djCTr7kxwbMJ/jA0meOhIj\nNia2UykztNVeH2JpvRGSZfUhljWYcmldiNa6IEGfvDciCKcjFRMKpdRPgaVlDn1Ya33vFC9zudb6\nmFKqFfiJUmqf1vrhcic6InInwPbt2+fAgX2KFAowcBjOvn7evrLS1If91C+v57zl5ZP6EpkcXYMp\njg8k6RpIcXzQiEjXYIqDPXEePdDHcJmozqZogKWOcLQ5pXe7rS4ow1yCsACpmFBora+Zg2scc8pu\npdQ9wMVAWaGoGsNdkM8suKGnShIJ+IqrtMZjOJXlxGCKrsEUJ4ZSxe2TQymOD6Z4qnOA/njmlJ8L\n+Cza6oK01ZpY0JbaIG11IVprg7TWBWmtNdsNEREUQZgvFuzQk1IqClha62Fn+zXAHVVu1qmIvXhZ\nTMaynw1jXjr0ks7l6R5KF4WkezhN95ARk5NDaZ7vGuKXL6ZHLQN28duKlpogLXUhWmqMiLTUGJuV\nZqdsrQ3SVBOQpcGCMEuqtTz2JuBzQAuwQym1W2t9rVKqHfiy1vp6oA24x3lq9AHf1lo/WI32TogI\nxYwJ+myz0qoxMuF5iUyO7qE0J4ccMRlO0zOcpns4Rc9wmqOxBE8eiRFLZMrGPkYCNk01ARqjQZqj\nARqjARprAjRFAyyJBIrHGiMBlkT91AR90lsRBA/VWvV0D3BPmfrjwPXO9kFg6zw3bfrEOkBZUL+y\n2i1ZtEQCPtY0+4yn1gRk8wX64xl6htP0jBgx6R1J0zeSoT+eoXfE9F6e6xqiL54hM06gvN9WLIkY\nQWmI+FkSCbAkGmCJs90QMdsNEb+zHaAu5Ju/bHZBmGekTz5bYh1QvwLsucsRFmaG37ZoqzNzG5Oh\ntSaeydM/kqEvniaWyNAfzxKLZ+hPZOgfMeVAIsP+7hFi8QwDyWzR1LEctSEfDRE/9WE/DeEA9WE/\ndWF/sc77qQv5qQv7qA+bIbo5z3AXhDlEhGK2LMB3KITJUUpRE/RRE/SxqmnioS+XQkEznM4xkMgw\nkMgS85SDySwDiSyDSfOJJTIcH0wy5Ox7TR/LURP0URfyUesIiJnj8VHnlLXF0nxqgmaIzGz7iAZ9\n8i6LUDFEKGZLrAM2XVftVgjzgGWpYo9gddPUf05rTTKbZyCRZSiVZTCRZSiVK4rIYDLLcCrHUCrL\nkLN9YjDFge4cwymzn5ugJ+MS8FnUOqIRDfqoCdql7YCPSNCmJugjEvARDdqmDNiEA2Y7ErCdj8+p\ns8VoUgBEKGZHJg7xbulRCBOilHJuxD7aCU/757XWpLIFIxrpHMOpHCOpHCNpIzjxtLtf+sSdsm8k\nw5G+BPFMjng6TzyTKzvhPx4+SxH224QCNiG/RchnhCXkc+p8FkG/KUN+m6CnDPotgj6bgM8i6LOc\n0uwHbHM8YFvFfb9Tmm2F37bwWUoWFlSQQkGTLRQm7fEuSqHoGkzxrd8cZm1TlLUtUdpqQ5UxvRs4\nYkoRCqGCKKUIO0/+rbO8lis68UyOhCMciUyehFMmM6Yu6WyncnmSmQLJbJ5kJkcqWyCVy5PK5hlM\nZunO5knnCqSyps7dnkIHaMoEbAufrfBZioDPwmeZfVdIfMXSnGNbCp9lYTvbtqWwlcK2ndJSWEph\nW8Z1WSlTbynz/9oqboOlzHGlQIFTlvaZiohpjTYFGo3WUPBsa60paCho95imoDX5gjmWL2jyWlMo\naHIF91jp49bl8mY7V9Dk8gXyBU02X3D2zXY2XzDbjjhMNOfmZVEKRd9Img/fs7e4H/bbrG6KsLY5\nyuqmaDEIaG1zlJba4MyfWGRprHCa4RUdxn9fctbk8gXSOfeTJ50tkMkXnDJfPJbNmfpsvkAmVyCT\n12RyBXJunXuDy5kbXiZvjpmbnSbvueG5N8G8M9SXK5RurvlCoXRz1ZpCwUQZuzdl90adN3fwYp3G\nnFu8qWNu3u6Nf6qUhMYRIVdsHDGylEJhhjctj0DZVum4zxE6yyoJns92hdGcE/bb2JbCb5cE0wio\nhd8VV6f0F0uz/SefGL/9i1Iozltez31//Tsc6o1zsDfOoR6T4fDCyWF++vzJUd2sSMBmdVOUNU2R\nYrmqyQjJ0rpJeiJFoVhb2V9IEE4zfLaFz7aIBqvdEmGq/MkExxalUACO02mYy9Y3j6rP5QscH0hx\nqC9ezHI43BfnxZPDPPR8N5l8aW19wGeZECAnu2F1oxGTVU0RljeECcU6IFg3r/bigiAI882iFYrx\n8NmWCexpimBeDC+RL2iODxj77Y6+OEf6THm4L8FjB/tIZPLFc5WCr4ceZ7ndzGfu2u3kN4RZsSTi\nWHKHxC1VEIRFwRknFBNhW6poKTG2J6K1pnckw5F+Ixyd/Uk2PN7HYbWcJ4/EuH9P1ykTQ211QZY3\nlMRjeUOY5UvCrHBK8SASBOF0QO5UU0QpRUutMZu7cHWjmcnaeYKlF72WR679HXL5Al2DKZPdEEty\nNJbkaCzB0ViS3Z0D/HBv1ylL0Boiftrrw0URaW8IFYfMljeEJaJUEIQFgQjFTBk5CblUccWTz7Ym\nNLjLFzQ9w2mODRjxcIOAjg+kONKX4LGX+k5xSfVZiqX1JklumSdZrhgMVB+iMRqQdeaCIFQUEYqZ\nMs0VT7Zz019aH+LC1eXPGUxm6XJCgI4NpOhyxWQwxZNHYpwYPLVXEvBZRdFYVh8uBgN5y6ZoQAzr\nBEGYMSIUM6UC71C49hBnL60re7xQ0PTG03QNpOgaNIlyJwZNENCJwSS/PdRP93DqFDGxnKjSVicQ\nqLUuSIsTANTqBgPVmRwHsWwQBGEsIhQzJdYBKGiYP3txy1JOwluIrSsbyp5TKGj64hlODpXS5Xqc\nIKDuYVP39NEB+uLlsxsaowFanbmY4qemtN3qBANJZKkgnDlUK7joU8ANQAZ4CXiH1nqgzHnXAZ8B\nbEyg0cfntaHjsedu+PVnAA2fuxCu/ghsubnarQKMmLg39fEyr8FkN/SNZDxhQCknDMjkOPQMpznY\nE6dnOD3q3RKXgG3RVBOguSZIs1vWBmmKBoopc+7xJZGA2GgLwmlMtXoUPwE+pLXOKaU+AXwI+Cvv\nCUopG/gC8GrgKPC4Uuo+rfVz895aL3vuhh+8F7JJsz/YafZhwYjFVPDbVnHOZCK01gwlc8UgoJ4R\nE1faO2KCgHpHjLg81zVE30imrMupUtDoJMk1RYM01gScpLmgU2dCgtykufqw5DMIwkKiWgl3P/bs\n7gTeVOa0i4EDTtIdSqn/BG4EqisUD91REgmXbNLUn0ZCMVWUUtRH/NRH/KxvndgcqFDQDKWyjoBk\niulyfSNpeuOm7BvJ8NzxIfpG0gylTs3CBjOn0uAkzLnxpI1ObKm3LCXMmSAgWUosCJVhIcxR/CFw\nV5n65UCnZ/8o8PLxLqKUug24DWDVqlVz2b7RDB6dXv0ZhGUpGpyo0PVTsDnN5gvE4hn64hkTV5rI\n0D+Spt+p63c+h3rjPHF4gFgiM67bpVJQF3LiScN+6iMBU3oS5urC/mJdnaeMBmyZbxGECaiYUCil\nfgosLXPow1rre51zPgzkgG/N9vu01ncCdwJs3759Dk2Ox1C/wgw3lasXpoXftmitC9E6hehSKA2D\nxRJGVAY9KXMDbsqckzQ3kMxyuC/OYNKEAU3kpuyzFHVhP3Uhn1O6IuIkzjn1pcS5UrpcXchPTcgn\nQ2XCoqZiQqG1vmai40qpW4HXAldrXdaw9xjgXVK0wqmrLld/ZPQcBYA/bOqFiuIdBltDdMo/Vyho\nRjI5Bj1RpWM/w6ksQ8lSytyJoVSxPpU9dTJ/LNGATY0TWeqNKK0J+ky9U0aDpfqop4w66XNhv/Ru\nhIVHtVY9XQd8EHil1joxzmmPAxuUUmsxAvEW4K3z1MTxcechHrrDDDfVr1hQq56EU7EsRV3I9BRm\nspg5kzPpcm586XCqFFE65JQjTsrccNo9nqNrMFVKn5tispxSEHWiSqNOfKkbWRoJ+oj4TbxpOGA7\nMaalCNOwf3SMadhvF7Mnwn6JNRVmjir/MF/hL1XqABAE+pyqnVrrdyml2jHLYK93zrse+DRmeexX\ntNZ/P5Xrb9++Xe/atasCLReEmVEoaBLZPHEnyjTuiSuNZ3KMpPPFuri77aTOxdOjSzeNbio52l68\nsaZhv4k2DfttE2Xqtwn7TYxpyGeOuRGnwWK0qVM6kaZuvKk36tQbbRrwWaOCcaSntLBRSj2htd5e\n7li1Vj2tH6f+OHC9Z/8B4IHpXn8wmeWpIzGWN4RpFmM9YQFgWao45NRW/sX7aZPJFYqi4caYJjI5\nJ7Y0b0p3O5Mn4cSVmk+heE4x1nSodCzlpNKlcvlpJblNRMBNVXMExBtx6gqKz1b43VQ228JfjDi1\nRqW52U5im6WcfSf9zWeVEuAsTwyqqXMT5EpRqN7oU8uNPB2TQmduH+Ye4k2qK9Wemojq/X/mpuKV\nts2Wm5jnRqC65xWcBD5vNGreTeHzRKG6caljo1Hzzn4u70SkFtxY1FJ0aq4YkVqKSp2IhbDqac45\n0p/gpn95FDB/OZc1hBx3VteZ1bi0uiZ7YvctnI6Yp/kADeV9KOcErU38qJuFncmdul2MMXViTb11\n3mPZgiabK0WcZnIFcoXC6Dzngi6eH8/kR2U/l25ybja0qSvFnepp97IWG5ZiVF64pcyikaLAOkJs\nj8ka91sTD0tWZeip0py3dZv+9Hce5PigY/k9kHQM9lKcHE6d8oS0JOJ3RKPkzup1aV1aH5LxXUE4\nTSh4nqqLT98F75O5k5mttcm/9mRkm+XXbl0pMxtKudku7rZJ0C7Vub0LRamb4e1xqDKZ2YrR+dlu\n78ayzDHbKvV8bMvN03Z7SxQztGczvLfghp4qTchvc83mtrLHsvkCJ5zcCOPUmuL4gDHYOxpL8NtD\nfae8CKYcU712RzS8ArK0zuy31gUJ+SXRThCqjWUpLBTyz3HuWJRCMRH+SXIjAOLpXFFEjDtr0ji2\nDqU42BPn0QN9DKdPfau4MRooWnu31bk230Fa60JF19bGSEDmTARBOK0444RiKkSDPta31rK+tXbc\nc4ZT2aJDa9dgipOO3ffJISMuT3cah9ax+Cxl3FnrSjbfra71t2MF7prqyXCXIAgLARGKGVLrvKE7\nkZikc3m6HXvv7qF00anVtfzu7E/wxOEY/WUEBUwPxWvx7bX8bi6WxvtIeimCIFQKEYoKEvTZkw5z\ngVnm6Lqw9oyx/O4eMg6th3rj9IykyeROfUvYthRN0ZLVd3M0YErH5rvJYwXeGA1IT0UQhGkhQrEA\nCPis4tLdidBaM5TKGVFxBMT99AyXHFtf6h4ZV1QAGiJ+mqLG8ruppmTv3VS0+nbswKPGmVViVAXh\nzEaE4jRCKVWMS13XMrHlt9aa4XSO3mHjxupaf/c5YuLW7e8e4TeHMsQS5RPvlDIRrUY8zDBXU43X\n6tuIiVsuiQSokzwJQVhUiFAsUpQq+Rud1TL5+bl8gVgi61h8m9wI1+47Fs8U6w/3JXjyyOSW3/WO\npbexHS+/XR82Bn+u9Xd9WHovgrAQEaEQAPDZVnHCHMafoHdxh8EGEhliiSyxeIaBZIZY3Fh+DySz\nxBz7776RDC/1jDCQMIZ5E1ET9HnyInxFAal37b8jo23A60KlbImgzxI/IUGoACIUwozwDoOtbpr6\nz2XzBYYca++BZLZo/W3yJHJOfaZ4zqHeOENOfTKbn/DaflsVhaOUHeFzPv7RZdCxBHePO/shv4iN\nIIxFhEKYV/y2RZOzEmu6ZHKFYl7EUCpXDCUaSrmZEiUbcPe8k0Opoi14PDOx0IBZQeaa99V68yNC\nPmoCzn7IR03QLuVJBErnRZxciahjCS7LloXFgAiFcNoQ8FlmCfAMRAaMj8/ImAwJN1di2M2NSGed\nXImSHXgskaEzlmDEtQefguC4hP02USdTIhIw4uLmR3jrQv7RGRLFbY8tuHss5FiEB2zp/QjzgwiF\ncMZgW6WEvNngzZYY8eRKJNJ5J1si58mOKOVIjKRzJJ2yeyhNIlv6mamk6I3FUjiiYXIjQsVsCauU\nKeGUISdLIjRetoQjPEHnZ0ZlSzjbbuaE36n3zdKETjh9WJRCkc4VGEnnqAkuyl9PqDKjsiXm6JqF\ngibt5EuMypPwZEYkMk6OhCdXwtR5MySc7awRpHSutJ/K5snkCzMSpXIox8I6aFtOxoQq5kz4bQu/\nz+RIBDzbfk+2hJtH4WZL2E4uhW0p/E7mxOgMilIWhZs74c2fsC2cnAnHadV1V/XkUHgzKGw1xsF1\njJMrOLkUSo3KnXCdX8f+v/Aydqm512nWmz0xNp/C61hbcH6olD/h5lVQdMb1ZlMUCpA7Ja+i9HGP\n5Ty5FPlCoWjRPhHVikL9FHADkAFeAt6htR4oc14HMAzkgdx4FrhjefHkMOf93Y+ckJhg0aSvrS5E\nW63Zb3X2W2qCBHyyJFOoLpalirGllaZcxoSbLZHOFUg7gjI2Y6KYK5Ev1Zv9Up6EW+dmTGSc+lS2\nQC6fI5vXxQyKTN4bqOP8jFOe6bkS881kU2nVikJ9DfAzrXVOKfUJAK31X5U5rwPYrrXunc7112/e\nqj/wr//NCdekb6jktVTuL2BTNOAIR8mkr60uSItj1tfqLBsN+sS3WBDmCzeQKFcwYuTNmXAT3Mbm\nTuTHPFEXn76dJ3L3KdxNhys91ZeOe5/4ofRkrzXFY3iOj7qjaIqxd9577ym9kjE9E/e4mzmhxvRw\nbKfn5E3i82ZQuCFFbmCRe8ztXbk9M5MCaGFZpUAjNwFwweVRaK1/7NndCbxpLq/fEPHzrleuO6W+\nUND0JzKcHEo5H2OFcXI4Rbez/9zxIXpH0pR7oGmI+Etur7VjzfnEpE8Q5hLLUgQsRQDp8VebhTCI\n/4fAXeMc08CPlVIa+KLW+s7xLqKUug24DWDVqlVlz7EsVVw1c257/bgNyhc0ffF00fnVFZSekZRT\n55j0DafJ5Mub9DW6hFNS6QAADkdJREFUJn01xgHWeCoFHZO+Ul1jNCA9FUEQFjQVEwql1E+BpWUO\nfVhrfa9zzoeBHPCtcS5zudb6mFKqFfiJUmqf1vrhcic6InInwPbt22c1nmZbyuk1hIDxBcVr0tc7\nnKbHKV1zvt6RND0jGQ72xOmLp8edRKwN+Yqi0hg1Bn3udlONMesrei2J+6sgCPNMxYRCa33NRMeV\nUrcCrwWu1uNMlGitjzllt1LqHuBioKxQVIPpmvQlMvmiOV/viPFT6htJ0+d4KvUOp+noLWVUjDef\nVxfy0VQTZEnEX3R9XeIRksaoMedrigZZEvVTE/TJMkZBEGZMtVY9XQd8EHil1joxzjlRwNJaDzvb\nrwHumMdmzilKKaLOG7urm6KTnp8vaAaT2aKQuAZ9/SMZ+uOluqOxBHuOGpO+bL68svhtRUMkQKNj\nyud1fV0ypq7BqasXB1hBEByqNUfxeSCIGU4C2Km1fpdSqh34stb6eqANuMc57gO+rbV+sErtnXfc\neY7GaIANUzjfa9LXHze24f3xLP3xdNG0L5Ywpn37u0eKZn4TOcDWhfzG7dURDuP66jHp8xj2eT+R\ngC09GEFYRFRr1dP6ceqPA9c72weBrfPZrtOZ0SZ9k/dYoJRZYUTEcX1NZI2gJLIMuqVj4He4L170\nV5pombvPUkURqQv5qHNExRj2lRxf60K+UXWuYZ8IjSAsLBbCqiehSngzK6bjAFsoGIFxHV6N+2vJ\nnM8Vk0HHvG8omeXYQJKhpNkut1LMi22pkutr8FQH2Bqv86tj3uca99UG/Y5pn09epBSEOUKEQpg2\nllXqvaycwc+nsvmiMd+QR0xcl9dhj3HfcMocPzaQYjg1XDTzm8xyAIyJoHFytakJ+ouOr9GgxwnW\nW+cY9hkXWB9Rx7AvGjCusLLaTDhTEaEQ5h3XyK518nyksmitSWULRlQc11dXQEbSOUYcS/GhVNa4\nvabzRSfYvpEMR/oSReO+6TjBBmyLcMAmGrCJeFxgowFf0fHVdYSNBGzCAR9hxxW26A7rcYj1HpPQ\nJWEhI0IhnHYoVfJFap3ltVwn2IQjGkUn2EyOkXSeZMYIjbufyLiusKY+mclzYihF0nGKdY9P16tI\nKQj5SgIS9FvGYnyUG6yp9zrBhhzn16ILrK/kAOu6vbrHStujXWElflaYDBEK4YzG6wQ7l2RyhaL7\n61hHWNfx1RUXrxusOa/k9prKmfNi8azjDlsg7ZSpXH7cJdHTwVJ4bMWNwPhtVbQUd23FjSuscpxh\nHZGxlHGOtUr1fkvhc5xh/Y4DrM+2HEdYj0Os4yRr2153WGu0W6xzzPJ4FHn9iYplcRvHMdZ1hl38\nvTSvT9Uor6sCZb2w8h7PrJKT7MTzhotSKPZ1DXPD535lMqBrgjTXBpzS7LfUmu1aeRFNqBDuE3t9\neHbZF5ORL+iicBj317zjAFsgk/fWm2OuS6zX/dW7nfY4xHpdYrN5Y4GeK2jPcbPtmvZl8yVX2IWC\na6LntRE3RnrGtM+1HzfW4qV6r2mfe8zFNfUDc85kuNbi4LEb91iMj7YeNz/htRTX2rUgd0wOtT7F\nxLDSLEqhqAn5aKoJcHIoxd5jg+Oa/AV8lhGQmkDRA6q5NuD4MZV8mpprgtSH/WL0Jyw4bEs58yLV\nbkkJrUt5B654ZAuF4hNsrqDJOULjPs2657vnFsbkKJR7Gh7rFJt3Mhxcp9jSDdVximW0i6zWo5/G\nixkR+tQbuPm9TJ1bUe7+rLUu+/Cpiv8ZK0Ce/VEZGIwSMNdV1rI8gudxmrUtVRS+0T2ukvOs2zMr\nOsx68j1sC67+xPh/potSKFYsCfPVd1xc3M8XNLGE4780nKFnJEXvcMbxYjKWGscHU+w5Nkh/PFN2\nRY37AlxTNFA09muKuqXjyeTZjsq7AMIZinJsrX22WbggnP4sSqEYi+1xjS1rU+ihUNAMJLNFU7/e\nEePBZCw00vQMm/LwkTj9I5lxV80EfFbRzG/UJ+L6MZWsM4y1RkDW/QuCsCA5I4RiOlge64yNbZOv\n30xl88bUzxGWvhFjoeF6M/WNpOlPZDnSn6B/JMNwOjfutWqCvqLXklu6guLWjfVoEsM/QRAqjQjF\nLAn5bZY3hFneEJ7S+ZlcgYGEERHXe8mUGfrHWGgc7ksQS2QYTo0vLralih5MDZEADWE/9RE/DWFX\nWEoeTF7Ppjox/RMEYYqIUMwzAZ9Fa53J7J4quXyBAccmwzXziyUyDCayDCRdXyZTd2Ioxb4Twwwm\ns4xM0HsBqA0aHyavmHjN/lyfpnqPV5PZ9knYkiCcQYhQnAb4bKs0xzINsvkCA4mS/9JgMuOIzame\nTAPJLAe6R4r16dzESxyDPqtk7OeIiOvHVDL5Mz5MXsM/95yaoE96NIJwmiBCsYjx25Z5l6R2egID\njh/TGGO/oVRJWFw/pqGkKQcSGTr7E8W6qayljwbskrmfa+zn8V1yjf6iQWP+Fy36NpXOiQZ9RPy2\nLF0WhAoiQiGUpejHNI0hMi9e4z/Xi2ms4d9I2tSZ0uyfGEwRT+eMh1M6N+WXiSJFA7+SkV80aBfN\n/SLufqC0HwmO79MUCYj7rCC4iFAIFWG2xn9gXl5KZvOMOCIST+c9Zn654vZI2ng0Ff2YHJHpGUmT\n6E+QSOeJOx5MU3GddfFZapR5X9gjJCH/aFO/otmfZ9vUOz5NAbvo5TTKu8lnSW9IWPBUTSiUUv8X\nuBEoAN3ArU5w0djz3g78rbP7Ua311+avlUI1Ucp969g3a/M/MMKTzhVGmfcZgSkZ+iUzeeIZYwbo\n1iczeRLZ/9fenYXGVYZhHP8/JmqkKtYNqhX1QlxQKVJ77UZbRRQFQRFFVNSL4o0iaqnVLqBW6YWK\nqOCV251Y3KsgXohQldaFViniVhSVulWbOpm8XpzvTE7S6UnSmcmZE58fDJnzzZnk/QiZN2d7ztjY\nrj0j/PLXHoYbY693krt00OABDOUhfykE8ODBvb8Ww/7GPS8E/RXD/8bym8aPt7Kb8kynASfXWrkq\ntyjWRcQKAEm3A/cBtxVXkHQksBJYSHbF/MeSNkTEbzNdrNWfpNaH8ZFzup950WiOFoL9mgw3RluB\nf7sbTYZTQxlu5KF/2et7Glk+UysIMAX+7Wlk+Uo7/85ymoYbWe5Svv5kJxxMR94wDswbSTEAsBUC\nmAXzFUMAB9M6xXDAPBBwMIX8DaaxAwfGYiMGByaE/00IA8wfB6SrvFvRE3nw34RgwDy/KR9XirfI\nxtU282l8TEZ/BQiOy3liLHJktBg5kiJLmimmJIJWtElE+0DAYuTJSHN8HEqZyhpFRPxZWJxD++iU\nJcDGiNgJIGkjsBR4sex7b/95V7fKNJuy/EP1sKHeBgHm8i2krHmMNZHich76l4cFZoF+wb8jzdbr\njRTk10hhf9l4ymkaHXueBwDu3t0cy3BqjtIYLTxvZjlOI4WMpjophgCKrJGgVkzTXmGAE3vL6fMO\nZ+uPf44bawUBpo+4Vn7UXoGAUXqL4SpVeoxC0lrgeuAP4Pw2qxwPfF9Y/iGNtftetwC3pMVdkr7s\nYqn7cjTw6wz8nJni+fS/2TanWTWfrfWez4n7ekHRw4xaSe/QPl1peUS8UljvHmAoIlZOeP+daXxN\nWl4B7I6IR3pW9DRI+igiFlZdR7d4Pv1vts3J86mHnm5RRMRFU1z1eeB1suMRRTuA8wrL84H3Oi7M\nzMymrLITxSWdUli8HNjWZrW3gMWS5kqaCyxOY2ZmNkOqPEbxoKRTyU6P/ZZ0xpOkhcBtEXFzROxM\np9FuSu9ZlR/Y7hNPV11Al3k+/W+2zcnzqYGeHqMwM7P6c0aBmZmVcqMwM7NSbhQdkHS/pB2SNqfH\nJVXX1C2S7pAUko6uupZOSFot6dP0+3lb0nFV19QJSeskbUtzelnSEVXX1ClJV0n6QtJoOkZZS5KW\nSvpS0nZJd1ddTze5UXRufUQsSI/Xqy6mGySdQHaG2XdV19IF6yLi7IhYALxKFhVTZxuBMyPibOAr\n4J6K6+mGz4ErgferLmR/SRoAngAuBs4ArpF0RrVVdY8bhbWzHriL9rEqtTLFqJjaiIi3IyK/deGH\nZNcW1VpEbI2ImUhS6KVFwPaI+Doi/gVeIjvtf1Zwo+jcsrQb4Nl0rUetSboc2BERW6qupVskrZX0\nPXAt9d+iKLoReKPqIgyYRtxQHfl+FJMoiyEBngRWk/2Xuhp4lOyPt69NMqd7yXY71cZkUTERsRxY\nnqJilrF3AkBfmUr0jaTlwAhZqkHfm2qcj/UnN4pJTDWGRNIzZPvA+96+5iTpLOBkYEuKXJ4PfCJp\nUUT8NIMlTksXomL6ymTzkXQDcClwYdTkQqhp/I7qagdwQmF5fhqbFbzrqQOS5hUWryA7KFdbEfFZ\nRBwbESdFxElkm8/n9HOTmMwUo2JqQ9JSsuNHl0XEP1XXYy2bgFMknSzpIOBqYEPFNXWNtyg687Ck\nBWS7nr4Bbq22HGujbVRMjT0OHAxsTFt9H0ZEreck6QrgMeAY4DVJmyNiScVlTUtEjEhaRpZFNwA8\nGxFfVFxW1zjCw8zMSnnXk5mZlXKjMDOzUm4UZmZWyo3CzMxKuVGYmVkpNwqzGSTpTUm/S6rFxZlm\n4EZhNtPWAddVXYTZdLhRmPWApHNTWOSQpDnpfgtnRsS7wF9V12c2Hb4y26wHImKTpA3AGuAQ4LmI\nqHXEi/1/uVGY9c4qsgygYeD2imsx22/e9WTWO0cBhwKHAUMV12K239wozHrnKWAFWbz5QxXXYrbf\nvOvJrAckXQ80IuKFdD/lDyRdADwAnAYcKukH4KaIeKvKWs0m4/RYMzMr5V1PZmZWyo3CzMxKuVGY\nmVkpNwozMyvlRmFmZqXcKMzMrJQbhZmZlfoPpKREw7KarQIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z2fx87mRf_ap",
        "colab_type": "text"
      },
      "source": [
        "可以看到，同一位置上，目标函数在竖直方向（ $x_2$ 轴方向）比在水平方向（ $x_1$ 轴方向）的斜率的绝对值更大。因此，给定学习率，梯度下降迭代自变量时会使自变量在竖直方向比在水平方向移动幅度更大。那么，我们需要一个较小的学习率从而避免自变量在竖直方向上越过目标函数最优解。然而，这会造成自变量在水平方向上朝最优解移动变慢。\n",
        "\n",
        "下面我们试着将学习率调得稍大一点，此时自变量在竖直方向不断越过最优解并逐渐发散。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nU7PqcToCKBc",
        "colab_type": "code",
        "outputId": "c3bba28f-17cb-4431-d125-13648f61c180",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        }
      },
      "source": [
        "eta=0.6\n",
        "show_trace_2d(f_2d,train_2d(gd_2d))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "epoch 20,x1 -0.387814,x2 -1673.365109\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEGCAYAAACgt3iRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de5wcdZnv8c/Tl7nkQiAkkMAMBiTq\nAgaODkGOuIsGISBsADUHVyXLYY03ZNeFo1x2ERE8KPpiRVkUFUU9LmRlWW4RDHnpouuihIsBhGAI\nsEkgEJKQC0nm0v2cP6pquqane9LVPTPd0/N9v17zqq5fV3f/Kpd+5vk9v/qVuTsiIiK1SNW7AyIi\nMvYpmIiISM0UTEREpGYKJiIiUjMFExERqVmm3h2ol2nTpvmsWbPq3Q0RkTHl4YcfftXdpxe3j9tg\nMmvWLFasWFHvboiIjClm9kKpdg1ziYhIzRRMRESkZgomIiJSMwUTERGpmYKJiIjUTMFERCSycglc\newRcvnewXbmk3j0aM8bt1GARkQFWLoG7zofeXcH+1rXBPsCchfXr1xihzEREBGD5FYVAEundFbTL\nHimYiIgAbF2XrF0GUDAREQGY0pGsXQZQMBERAZh3GWTbB7Zl24N22SMFExERCIrsp10HE/YN9idM\nC/ZVfK+IZnOJiETmLAR3uH0xLLge3jy/3j0aM5SZiIjE5XqCbb6vvv0YYxRMRETi8r3B1nP17ccY\no2AiIhKXC4OJMpNEFExEROL6h7ny9e3HGKNgIiISp8ykKgomIiJxOdVMqlHXYGJmN5nZK2b2RKxt\nqpktM7M/hdt9wnYzs+vMbLWZrTSzt8Vesyg8/k9mtqge5yIiTUKzuapS78zkh0DxRO6LgOXuPhtY\nHu4DnAzMDn8WAzdAEHyALwDHAHOBL0QBSEQkMQWTqtQ1mLj7A8DmouYFwM3h45uB02PtP/LAg8De\nZjYTOAlY5u6b3X0LsIzBAUpEpDJREFEBPpF6Zyal7O/uL4WPNwD7h48PBNbGjlsXtpVrH8TMFpvZ\nCjNbsXHjxuHttYg0B2UmVWnEYNLP3R3wYXy/G929y927pk+fPlxvKyLNJAomKsAn0ojB5OVw+Ipw\n+0rYvh7ojB3XEbaVaxcRSS4XDXMpM0miEYPJnUA0I2sRcEes/exwVtc7gK3hcNh9wIlmtk9YeD8x\nbBMRSU7DXFWp66rBZvYvwPHANDNbRzAr62pgiZmdC7wAROs/LwVOAVYDO4FzANx9s5l9CXgoPO4K\ndy8u6ouIVEZXwFelrsHE3T9U5ql5JY514NNl3ucm4KZh7JqIjFd5DXNVoxGHuURE6kcF+KoomIiI\nxKlmUhUFExGRuP7ZXMpMklAwERGJ689MFEySUDAREYnTMFdVFExEROK0BH1VFExEROLyujlWNRRM\nRETidNFiVRRMRETidNveqiiYiIjEKZhURcFERCROBfiqKJiIiMRpanBVFExEROL6Z3OpAJ+EgomI\nSCSf16rBVVIwERGJRFkJqGaSkIKJiEgkqpeAMpOEFExERCK5WGaihR4TUTAREYkomFRNwURExp+V\nS+DaI+DyvYPtyiVBu4a5qlbXe8CLiIy6lUvgrvOhd1ewv3VtsA/Q0VU4TgX4RJSZiMj4svyKQiCJ\n9O4K2gcMcykzSULBRETGl63ryrcPGOZSZpKEgomIjC9TOsq3R5lJulXBJCEFExEZX+ZdBpm2gW3Z\n9qA9CibZNg1zJaRgIiLjy5yF8K4LCvtTOuG064L2aJgrO0EF+IQadjaXmT0PbAdyQJ+7d5nZVOBW\nYBbwPLDQ3beYmQHfAE4BdgJ/7e6P1KPfIjIGdB4TbN80H/7q1kJ7tJxKRplJUo2embzb3Y9y92i+\n3kXAcnefDSwP9wFOBmaHP4uBG0a9pyIydnRvC7bFs7r6h7natWpwQo0eTIotAG4OH98MnB5r/5EH\nHgT2NrOZ9eigiIwB3duDbd/uge3RMJcyk8QaOZg48Asze9jMFodt+7v7S+HjDcD+4eMDgbWx164L\n2wYws8VmtsLMVmzcuHGk+i0ijS4KJoMyk1jNRMEkkYatmQDHuft6M9sPWGZmT8efdHc3M0/yhu5+\nI3AjQFdXV6LXikgT2R0Ocw3KTMIAkm1TAT6hhs1M3H19uH0FuB2YC7wcDV+F21fCw9cDnbGXd4Rt\nIiKD9ddMhhrmUjBJoiGDiZlNNLPJ0WPgROAJ4E5gUXjYIuCO8PGdwNkWeAewNTYcJiIyUBRM+oYa\n5lIwSaJRh7n2B24PZvySAX7q7vea2UPAEjM7F3gBWBgev5RgWvBqgqnB54x+l0VkzOivmRRlJvnY\nMJdqJok0ZDBx9zXAkSXaNwHzSrQ78OlR6JqINIPde8hMMu0KJgk15DCXiMiIijKTfF+h6A6xYa6w\nAO+ap1MpBRMRGX+iYAIDs5MosGTag63rwsVKKZiIyPgTFeBh4LUmuR6wFGRagn0V4SumYCIi40/3\nNmiZFDwuDibpFrB0sK+6ScUUTESkuZS7v3vEPRjmmjg92I9fuJjvC4JJKpybpAsXK6ZgIiLNI7q/\n+9a1gBfu7x4PKD2vB7WQSeFqTMWZSSoDKWUmSSmYiEjzGOr+7pGoXjKpRGYSDXNFmYlqJhVTMBGR\n5jHU/d0j0UyuifsF2wGZSW9YMwm/GhVMKqZgIiLNY6j7u0eiYBINcw3ITHohnYllJhrmqpSCiYg0\nj3mXBTe2iovu7x7ZvTXYRsNcpWZzqQCfmIKJiDSPOQvhPf9Y2J88o3B/98geM5OsCvBVaMi1uURE\nqjb9zYXH778JZr1z4PP9BfgSs7nyvZDKqgBfBWUmItJcNj9XePz6K4Of789MwgJ8qdlcKsAnpmAi\nIs1l07OFYLCjxO25oxWDJ5aqmUTDXCrAJ6VgIiLNZfMamP6WIKCUy0xaJgV3U8RK1ExaCjUTFeAr\npmAiIo1vT0ukxG1+FqbNhgn7wo5SwWQbtE4Gs2Cm16DZXMpMqqFgIiKNrZIlUiK5PtjyAkw9JLgo\n8fUSw1zd26B1r+Bxpq3MMFc0m0tL0FdKwUREGlslS6REtq4NZmRNfWNwHUnJzGR7kJlAkJmULMBr\nanBSCiYi0tgqWSIlsnlNsO3PTEoEk93boK1MZjJoarCCSaUUTESksVWyREokCib7vjGY+rtj4+Bb\n7w6ZmRQNc6kAXzEFExEZHUmK6HHv/ofBbcVLpEQ2r4HsxOCCxInTg1vy9rw+8Jju7UPUTIpXDVZm\nUikFExEZeUmK6MUmTgu27fsG29bJg5dIiWx6NhjiMitclFg81BUvwA/KTPpUgK+SgomIjLwkRfRi\nK2+Btr3hgqegYy7sd3jpQAJBZjL14OBxtMR8/MLFfA56dhSGuUpmJlkV4KugYCIiQ6t2eCouSRE9\nrns7PHU3HH4GZFqhcy68+Cj0dQ8+NtcHW54P6iVQWBU4nplES6m0lctMNMxVLQUTkbFuOL7sh3rv\naoen4vaaWbq9XHE98vQ9Qd3jyLOC/YPeAblueGnl4GO3rQunBR8S7PdnJiWCSbwAH2Um+XxQcNcV\n8FVpmmBiZvPNbJWZrTazi+rdn7oayS+XehgL51OvPg7Xl305y75Q/fBUpHt7MN22mKVKF9Hj/nAL\n7P0G6Dwm2O+YG2zX/m7wsZueDbZTw8wkqrXEL1yMVgyOF+CjzCTfG2xTGa0aXIWmCCZmlgauB04G\nDgM+ZGaH1bVTzfrlUu4zG/0343LvPRz9rsefOcDOzXDfpbV/2Zey9iH42bmw/cXSz+9peCrS1w23\nfiQ4/tjzYEonYEENxPOFDKGUbS/Bc/8Bc/5XUFAHmLw/7DOrdDCJX2MCQe2jfWoFmUkYTHI94eu0\nanA1muV+JnOB1e6+BsDMbgEWAH8s94K+XJ47HlsHXu6IAsMGt5kxoDn2PgesvYsjH7uMTC78R7p1\nLX13fIbHn3uZDQeehHkO8zyQ739sngcfuG+Ejwn3PQf5ouc89hx5jnz4UlpLfLl03/15/vCykbcM\nnsrgqTRumWDfwsep8HEqOCYftkfHuqWBgec987/v4q2PljjXdVt4qfNUzHtJ5XtJ5Xo4eO8s7akc\nluvGcj1YvhfLdUOuh3SuF3Ld/W2W64FcD3v911dJlzif3NLP8drOPjw7ATLteHZC4ScTPW4vfCkU\naX/635iy/EJSfbv6+52/83y27exh95+dCfk+rHsrqd1bSe1+Det+jdTurWHblgGPW5//ZdDfoj7m\n77mQbTu7yU2eSX7STHzSAYPvAhhqfeo2Jv3my6S2ryc/+UBeP+5Suv/s/cG/tZ4dpF99msymp0lv\nWkXm1adIb3qadKkL8kK+dS3bHv13ejuPw4f6wo7L9dD6zF20P/JdshseId8yGVomkerZMejQfHYC\nmzZvDqbhFom+98nnmHzPJ2ld8yu2z7+O7iPOgneGmUiul71vPh679xJem3Fc8AVepO3hW5joebYc\nejr5HYUayaSZXWRfeIAt23fHPgwmvPwn2jIT2Gz7QHj8lAnTyW3dwI5wP/vaZvYCtno7fTu6meBZ\n2vp2sXlHN7brdaYCr+dS9OzOsw+wY9duuneUqM+MUy3p8vmHefEFPWOQmX0AmO/ufxPufxQ4xt3P\nK/ea1pmzfeaifxqR/vym5Xw6Uq+OyHvXW5+n6CNNH2lypJjELtI2+N+QOzhGqsRzo2m3Z9lJKztp\nY5e3spNWdtHKkbaadusddHyfp9hJK3vZrhLvVvC6t7KViWz1ibzF1sa/04a0xSexwfdhg0/t/5lh\nmzkz/WtarVDs7fU0z/iBTLGddFjh39Iub+EZ7+CZfAfPeAcfz9zNNNs26HPyDikLzudRP5Rf5+bw\n6/xbWemH8L7Uf/G5zBIOsFd50adxQ99pTLXtfCRzP/vbazybn8nNuRO5LffnzEs9zNXZ7zHBemJ9\nS5Emzwu+Pxf0fpJH/E0lztT5YuaHLMos46rev+K7uVMHHXF86jF+2PJVruz9MN/LvW/Q8/e0XEwv\naU7vuXJA+0fSy7gy+wOO6/4n1vl+/e3fz17DAbaJk3uu7m/7afZKstbHB3suB+C01G/5Zsu3mNd9\nDc/6gfxt+jY+m72NQ3b/hH3ZxkNtn+Ifes/h/tzbeLDtM3yu92Msyb27xPmNXy985dSH3b2ruL1Z\nMpOKmNliYDFAy4xDg7YR+JwDrHQgcYcv932YHCnypMKtDXzspdqD/cLjsN1T5MLjorbvtXyN/ey1\nQZ/9ik/hUz1/R8ZyZMiRJk+WPtLkyZDr/0lbYT9Njkzs+bTlyIavjdoWpe8r++fwrdzp9HiWHjL0\nkqGHbP+2O9r3DD3h4/6tF46/o+USDrDNg957g+/D2b2XMIFu2uhmgnXTTjcTCLe2m3a6aaeHdova\ndwdb66aNwYEEIE2e23J/EQQKJvKaT2JbGDReYyLbfBJbmUhv7L/OAy3n08Hgv/MXfV/O7r2IGWxh\nhm3u/9mfLcywTRyRep7ptrVkP7KWYzbruTd/DLf4e1jlnTyT72Ad0/HY6PTGvr35cmbgl/1Ob+Ef\n+87hRZ/OcanHeVfqcf4ucxt/bz9jp2dpIUfGgusnOuxVrsz+ADP4Vf5ILupbzAP5OcFnGNztx5Hq\ngwszSziATbzIvnytbyEv+1SuyX6bf235It/O/SXX5d4/4M/kM+nbWZRZxnf6TuX7+VNJlfiP9oAf\nxa9yR/K3mX/jjvxxbGJK/3NvsrUcnnqBy3sXDXptFLyOTj3Di/lCMJllG3jGOwccv4kpvJU1/W2T\nw18SdtJOyqDbgoyo3XpoJQjmfWTwMKPNki/Z91qkDHJj9PKVlkzzZybHApe7+0nh/sUA7v5/y72m\nq6vLV6xYMTIduvaIcPy8yJRO+OwTI/OZkWj8Pj40lG0vf5FXrUb6XEfqfIaz37X0sa8HrtyP0uOt\nBpcP/sWg5OcvvyKoS0zpCIraxZ+7czOs+RXccR70vj74PSbNgAtX7fmz4nZvg/suhkd/AjPeCoe/\nH1Z8v/DnetD/hHOWMmTatvEZuOFY+B8fgdO+UWhf9gX47TfhglWFKb6RfA6uPiiY4fW+rwdtuT64\nagYc+2l47xcLx/78oqB/l4Q1nv/8Biy7DC5eD62T4HffgZ9/Dv7PGtj9GnzzbXDGd+DQE+CaN8Ip\nX4O5H0v259LkzKxkZtIUBXjgIWC2mR1sZi3AWcCddevNvMsGj42XW/5huM1ZGHyJRYXOKZ0jF0hg\n5M91pM5nOPtdSx8zLcnWnir3+Z99Igg8n32i9OdOmApHnAm9O0u/x46XK/usuLa9YMH1cNa/wObn\nYfnlAwP0S4/C4/869HtMfxMc/TF45Eew4fGgLZ8PXnfoCYMDCQTTdju6Bhbho2nB0TUmkYnToGd7\nIdB3bw/qaC1hrSfTFmx7dxauKUllVICvQlMMc7l7n5mdB9wHpIGb3P3JunUo+s+8p98WR/LzR/Oz\nYGTPdSTOZ7j7XUsf511WOrMZiV8+pnSUycgqDFylvOUUaJscfGnHRbPK9vTncvznYeWtcO/FsOgu\neOE3sG09vHeIGWmdx8AD1xQWbeyfFnzIwOOiJVV2vAL7vCHIpqIbY0HhF4q+3QNnc+mixcSaIpgA\nuPtSYGm9+9FvNL/Q622snmuj9Hs0f/kYqcC17aXS7ZVMIW7fB959CSy9MBha2rkJsMKU3VI65wZT\ni9c/DIccH5sWXJyZROtzbQyCSXyRR4hlJrsK15noosWqNE0wERnTRiuwjVTgqjXjaZkMWBhIABx+\nfmEwDFiqbx1HB8ev/X0hmGQnwOQZA4+Lhsmia03iizzCwMwkqh/rtr1VUTARGW9GInDVmvH88koG\nTUIYapisbQrsd1ihbrJ5TWG14Lj+zCQeTGLX3cQzk6hOMiCYKDOpVLMU4EWknmqdKFHNQpCdc4Mr\n9fP5cOn5gwcfMzHKTMIlVeI3xoIgm4HBNRMV4BNTZiIiw6OWjKeaYbLOY+DhH8ArTwarBb9l8IWP\nZNugdUohM9m9bWCRPhvLTKIhr3Q2yHAsrWGuBJSZiEj9VTNVuzNc9PGJ2wauFlxs0vRYzWR76WGu\neGYSLUqZSqsAn4CCiYjUXzXDZFMPgQnTCgtqFl9jEpm4X2Hl4OLZXFEA6901cJgLgrqJMpOKaZhL\nRBpD0mEys2Coa9U9wf5QmckrT0GuN7g3SqmpwX27IRfWT9JRZpLRbXsTUGYiImNXprXw+Psnll72\nf+J+wTBX8V0WoUxmEgYTSykzSWDIYGJme5nZoNzRzOaMXJdERCqwcklwJ8ZIufvITNovWHfr9XAx\nzj3VTDTMVZWywcTMFgJPA7eZ2ZNmdnTs6R+OdMdERIa0/IrgFr5xpW4MFk0Pjq6SjwcTsyCg9O4K\nhsEgFkxUgE9iqMzkEuDt7n4UcA7wYzM7I3xuJFZuFxGpXKXXpkTrc20O1++K10ygcOve/uVU4jUT\nZSaVGqoAn3b3lwDc/fdm9m7gbjPrpKL7E4qIjKBKr02JroLftDrYFt95MtserBpcPDXY0irAJzBU\nZrI9Xi8JA8vxBLfDPXyE+yUiMrRKr02J1ueKVhZumzLw+UxbsKhkrjgz0UWLSQwVTD4JpMzssKjB\n3bcD84G/GemOiYgMqdJrU6LMpFTNBIIA1BcGE0sXVgzWMFciZYe53P0PAGb2hJn9GPgq0BZuu4Af\nj0oPRUTKqeTalJYJ0DKpMCRWqmYSTQ2OshJQAT6hSq4zOQboBH5LcEfDF4F3jmSnRESGVTSjK5Ud\neG0KDMxMoplcEGYmCiaVqiSY9AK7gHaCzOQ5d1dVSkTGjmhGV/wui5Fse+HmWPHMxFIKJglUEkwe\nIggmRwPvAj5kZnu4sbOISAOJMpO2vQY/F00NzvUUZnKBaiYJVbI217nuviJ8/BKwwMw+OoJ9EhEZ\nXvHMpFiUmQwa5lLNJIk9ZiaxQBJvU/FdRMaOaEZX65TBz8Uzk7Qyk2ppoUcRaX7RtSaJMhMV4JNQ\nMBGR5hdlJkPWTHohHRv5VwE+EQUTEWl+Lz8RbFfeCtceMXBl4f6pwd0lMhMNc1VKwUREmtvKJfCf\n3yjsFy9VHy1D371DBfgaKJiISHNbfkWQecTFl6qP1vfq3hZkIxFlJokomIhIc9vTUvVRZrJ72+DM\nRDWTijVcMDGzy81svZk9Fv6cEnvuYjNbbWarzOykWPv8sG21mV1Un56LSEMqXpK+uD2emcSDiSmY\nJNFwwSR0rbsfFf4sBQhXLz6LYPn7+cA/m1nazNLA9cDJwGEEV+gfVu6NRWSc2dNS9VFm0rNj4Gwu\nDXMlUskV8I1iAXCLu3cDz5nZamBu+Nxqd18DYGa3hMf+sT7dFJGGEq0qvPyKYGhrSkcQSKL2eKBR\nAb5qjRpMzjOzs4EVwAXuvgU4EHgwdsy6sA1gbVH7MaXe1MwWA4sBDjrooOHus4g0qqGWqi8bTHTR\nYhJ1GeYys/vD+6QU/ywAbgDeCBxFsBbY14frc939Rnfvcveu6dOnD9fbishYlokHE60aXK26ZCbu\nfkIlx5nZd4G7w931BPdViXSEbQzRLiIytGxb4bFWDa5awxXgzWxmbPcMILx0lTuBs8ys1cwOBmYD\nvydYIn+2mR1sZi0ERfo7R7PPIjKGZYYa5lIwqVQj1ky+amZHAQ48D3wcwN2fNLMlBIX1PuDT7kF1\nzMzOA+4D0sBN7v5kPTouImNQPDPRbXur1nDBxN3L3ivF3a8CrirRvhRYOpL9EpEmVa5mogJ8Ig03\nzCUiMqoGZCbxixZVgE9CwURExrchMxPVTCqlYCIi41sqVchIUkU1EwWTiimYiIhE2UnxbC4c8vm6\ndGmsUTAREYnqJsWzuUAzuiqkYCIiEi32WLxqMKgIXyEFExGRaH2u4gI8qG5SIQUTEZGSwUTDXEko\nmIiIlC3Ao2GuCimYiIhEBfhU0arBoGGuCimYiIhkhqqZKDOphIKJiEi2xGwuFeATUTARESmZmagA\nn4SCiYhIyYsWNcyVhIKJiEjJixZVgE9CwURExreVS+CRHwWPf/KBYB+UmSTUcDfHEhEZNSuXwF3n\nQ++uYH/HhmAfIDsh2CozqYgyExEZv5ZfUQgkkd5dQbsK8IkomIjI+LV1Xfl2DXMlomAiIuPXlI7y\n7f0FeAWTSiiYiMj4Ne+ywiKPkWx70K6LFhNRMBGR8WvOQjjtOpjSCViwPe26oD2qmSiYVESzuURk\nfJuzMPgpFmUmKsBXRJmJiEgpKsAnomAiIlKKCvCJ1CWYmNkHzexJM8ubWVfRcxeb2WozW2VmJ8Xa\n54dtq83solj7wWb2u7D9VjNrQUSkVirAJ1KvzOQJ4EzggXijmR0GnAUcDswH/tnM0maWBq4HTgYO\nAz4UHgvwFeBadz8U2AKcOzqnICJNTQX4ROoSTNz9KXdfVeKpBcAt7t7t7s8Bq4G54c9qd1/j7j3A\nLcACMzPgPcDPwtffDJw+8mcgIk1PBfhEGq1mciCwNra/Lmwr174v8Jq79xW1i4jUxqLMRMGkEiM2\nNdjM7gdmlHjqUne/Y6Q+dyhmthhYDHDQQQfVowsiMlakFEySGLFg4u4nVPGy9UBnbL8jbKNM+yZg\nbzPLhNlJ/PhSfboRuBGgq6vLq+ifiIwXKsAn0mjDXHcCZ5lZq5kdDMwGfg88BMwOZ261EBTp73R3\nB34JfCB8/SKgLlmPiDQZrRqcSL2mBp9hZuuAY4F7zOw+AHd/ElgC/BG4F/i0u+fCrOM84D7gKWBJ\neCzA54G/N7PVBDWU74/u2YhIU1JmkkhdllNx99uB28s8dxVwVYn2pcDSEu1rCGZ7iYgMHxXgE2m0\nYS4RkcagAnwiCiYiIqVomCsRBRMRkVJUgE9EwUREpBRlJokomIiIlKICfCIKJiIipaRSgCmYVEjB\nRESknFRaw1wVUjARESknlVEBvkIKJiIi5aQyGuaqkIKJiEg5llYwqZCCiYhIOaqZVEzBRESkHAWT\niimYiIiUowJ8xRRMRETKUc2kYgomIiLlpBRMKqVgIiJSTiqjmkmFFExERMpRAb5iCiYiIuWoAF8x\nBRMRkXJUgK+YgomISDkqwFdMwUREpBwV4CumYCIiUk4qrZpJhRRMRETK0arBFVMwEREpx1Ia5qqQ\ngomISDnKTCqmYCIiUo4uWqxYXYKJmX3QzJ40s7yZdcXaZ5nZLjN7LPz5duy5t5vZ42a22syuMzML\n26ea2TIz+1O43ace5yQiTUgXLVasXpnJE8CZwAMlnnvW3Y8Kfz4Ra78B+BgwO/yZH7ZfBCx399nA\n8nBfRKR2GuaqWF2Cibs/5e6rKj3ezGYCe7n7g+7uwI+A08OnFwA3h49vjrWLiNRGBfiKNWLN5GAz\ne9TM/sPM3hW2HQisix2zLmwD2N/dXwofbwD2L/fGZrbYzFaY2YqNGzcOe8dFpMkoM6lYZqTe2Mzu\nB2aUeOpSd7+jzMteAg5y901m9nbg383s8Eo/093dzHyI528EbgTo6uoqe5yICKACfAIjFkzc/YQq\nXtMNdIePHzazZ4E3AeuBjtihHWEbwMtmNtPdXwqHw16preciIiEV4CvWUMNcZjbdzNLh40MICu1r\nwmGsbWb2jnAW19lAlN3cCSwKHy+KtYuI1EarBlesXlODzzCzdcCxwD1mdl/41J8DK83sMeBnwCfc\nfXP43KeA7wGrgWeBn4ftVwPvNbM/ASeE+yIitdOqwRUbsWGuobj77cDtJdpvA24r85oVwBEl2jcB\n84a7jyIiWjW4cg01zCUi0lBUgK+YgomISDmpDHi+3r0YExRMRETK0UWLFVMwEREpZeUSeORm6N0J\n1x4R7EtZdSnAi4g0tJVL4K7zoXdXsL91bbAPMGdh/frVwJSZiIgUW35FIZBEencF7VKSgomISLGt\n65K1i4KJiMggUzqStYuCiYjIIPMug2z7wLZse9AuJSmYiIgUm7MQTrsOpnQCFmxPu07F9yFoNpeI\nSClzFip4JKDMREREaqZgIiIiNVMwERGRmimYiIhIzRRMRESkZubu9e5DXZjZRuCFUfioacCro/A5\no6XZzgea75x0Po1vLJ/TG9x9enHjuA0mo8XMVrh7V737MVya7Xyg+c5J59P4mvGcNMwlIiI1UzAR\nEZGaKZiMvBvr3YFh1mznA65fVoMAAAN+SURBVM13Tjqfxtd056SaiYiI1EyZiYiI1EzBREREaqZg\nMgrM7HIzW29mj4U/p9S7T8PBzC4wMzezafXuSy3M7EtmtjL8u/mFmR1Q7z7VysyuMbOnw/O63cz2\nrnefamFmHzSzJ80sb2Zjdkqtmc03s1VmttrMLqp3f4aTgsnoudbdjwp/lta7M7Uys07gROC/692X\nYXCNu89x96OAu4FmuAPSMuAId58DPANcXOf+1OoJ4EzggXp3pFpmlgauB04GDgM+ZGaH1bdXw0fB\nRKp1LfA5YMzP4HD3bbHdiTTHOf3C3fvC3QeBMX2/WXd/yt1X1bsfNZoLrHb3Ne7eA9wCLKhzn4aN\ngsnoOS8ccrjJzPapd2dqYWYLgPXu/od692W4mNlVZrYW+DDNkZnE/W/g5/XuhHAgsDa2vy5sawq6\n0+IwMbP7gRklnroUuAH4EsFvvF8Cvk7wH7xh7eF8LiEY4hozhjofd7/D3S8FLjWzi4HzgC+Mager\nsKdzCo+5FOgD/t9o9q0alZyPNC4Fk2Hi7idUcpyZfZdgXL6hlTsfM3srcDDwBzODYPjkETOb6+4b\nRrGLiVT690PwpbuUMRBM9nROZvbXwKnAPB8DF5Ql+Dsaq9YDnbH9jrCtKWiYaxSY2czY7hkExcQx\nyd0fd/f93H2Wu88iSNXf1siBZE/MbHZsdwHwdL36MlzMbD5BTesv3X1nvfsjADwEzDazg82sBTgL\nuLPOfRo2ykxGx1fN7CiCYa7ngY/XtztS5GozezOQJ7gtwSfq3J/h8C2gFVgWZpAPuvuYPS8zOwP4\nJjAduMfMHnP3k+rcrUTcvc/MzgPuA9LATe7+ZJ27NWy0nIqIiNRMw1wiIlIzBRMREamZgomIiNRM\nwURERGqmYCIiIjVTMBFpMGZ2r5m9ZmYNf3GrSETBRKTxXAN8tN6dEElCwUSkTszs6HDxzzYzmxje\nr+MId18ObK93/0SS0BXwInXi7g+Z2Z3AlUA78BN3H7NL7cj4pmAiUl9XEKzZtBs4v859EamahrlE\n6mtfYBIwGWirc19EqqZgIlJf3wH+kWDp+6/UuS8iVdMwl0idmNnZQK+7/zS8P/hvzew9wBeBtwCT\nzGwdcK6731fPvorsiVYNFhGRmmmYS0REaqZgIiIiNVMwERGRmimYiIhIzRRMRESkZgomIiJSMwUT\nERGp2f8H64Ezd+kPI0wAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xFHdzmvLgIUg",
        "colab_type": "text"
      },
      "source": [
        "###7.4.2. 动量法\n",
        "动量法的提出是为了解决梯度下降的上述问题。由于小批量随机梯度下降比梯度下降更为广义，本章后续讨论将沿用“小批量随机梯度下降”一节中时间步 $t $的小批量随机梯度 $\\boldsymbol{g}_t$的定义。设时间步 $t$ 的自变量为 $x_t$ ，学习率为$\\eta_t$。 在时间步 0 ，动量法创建速度变量 $v_0$ ，并将其元素初始化成0。在时间步 $t>0$ ，动量法对每次迭代的步骤做如下修改：\n",
        "$$\n",
        "\\begin{split}\\begin{aligned}\n",
        "\\boldsymbol{v}_t &\\leftarrow \\gamma \\boldsymbol{v}_{t-1} + \\eta_t \\boldsymbol{g}_t, \\\\\n",
        "\\boldsymbol{x}_t &\\leftarrow \\boldsymbol{x}_{t-1} - \\boldsymbol{v}_t,\n",
        "\\end{aligned}\\end{split}\n",
        "$$\n",
        "其中，动量超参数 $\\gamma$满足$0 \\leq \\gamma < 1$。当 $\\gamma=0$时，动量法等价于小批量随机梯度下降。\n",
        "\n",
        "在解释动量法的数学原理前，让我们先从实验中观察梯度下降在使用动量法后的迭代轨迹。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LLPboyuPCzok",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def momentum_2d(x1,x2,v1,v2):\n",
        "  v1=gamma*v1+eta*0.2*x1\n",
        "  v2=gamma*v2+eta*4*x2\n",
        "  return x1-v1,x2-v2,v1,v2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QnhJ_KiwDZ9M",
        "colab_type": "code",
        "outputId": "8ea35ac9-4397-42b1-ee7e-3555b04eea34",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        }
      },
      "source": [
        "eta,gamma=0.4,0.5\n",
        "show_trace_2d(f_2d,train_2d(momentum_2d))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "epoch 20,x1 -0.062843,x2 0.001202\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEGCAYAAAB7DNKzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOydeZhcVYG+33Nv7dX7nnSnswdIIAES\nArIjW0QREERQFAFFHR2XGcdlnEEGN9SZnwKiEhFZVBQXBARUNtkDJJCEsCYk3el0Oumturu69uX8\n/jj3VlV3qjud9FK9nPd56jn3nnur6gSS+9XZvk9IKdFoNBqNZiiMQjdAo9FoNJMbLRQajUajGRYt\nFBqNRqMZFi0UGo1GoxkWLRQajUajGRZHoRswHlRVVcl58+YVuhkajUYzZdiwYUOnlLI637VpKRTz\n5s1j/fr1hW6GRqPRTBmEEM1DXdNDTxqNRqMZFi0UGo1GoxkWLRQajUajGRYtFBqNRqMZFi0UGo1G\noxkWLRQajUajGRYtFJOFzffAjw6Ha8tUufmeQrdIo9FogGm6j2LKsfkeeODzkIio894WdQ6w/OLC\ntUuj0WjQPYrJwWPXZUXCJhFR9RqNRlNgtFBMBnp3HVi9RqPRTCBaKCYDpQ1DXJDw8Negv2NCm6PR\naDS5aKGYDJx+DRjOgXUOD8w9EV68BW48Ep74LkT7CtM+jUYzo9FCMRlYfjHMOQ6EAQgonQPvvwmu\neBD+5QVYdDo8+X24YQU8fzMkooVusUajmUHoVU+TBcOA+pXwiUcH1lcvgYvvhNaX1eT23/8Tnv8p\nnPZ1WH4JmPp/oUajGV90j2KyEGiC8nlDX68/Gj72F/jYfVBUA/d9Fn52PLzxAEg5Ua3UaDQzkIIK\nhRDiNiFEuxBiyxDXhRDiRiHENiHEZiHE0RPdxgkhlVArnIYTCpsFp8InH4eL7wIk/P4yuPV02PHU\n+LZRo9HMWArdo7gdWDPM9fcAi63X1cDPJqBNE0/vLpCpkQkFgBCw9P3wmefh/T+B4B6441y46wLY\n/cq4NlWj0cw8CioUUsqngO5hbjkPuFMq1gFlQohZE9O6CSTQpMqRCoWN6YCjPwr/+jKc9R3YvRHW\nngp/+Dh0bhvbNmo0mhlLoXsU+6MeaMk532XVTS8OVihsnB44/nPwhY1w8lfg7X/Azavh/s9Db+tY\ntVKj0cxQJrtQjBghxNVCiPVCiPUdHVNsg1qgCUwXFI+ys+QphXd/QwnGMZ+Ajb+Fm46Gf/w3hIfr\nuGk0Gs3QTHahaAXm5Jw3WHX7IKVcK6VcJaVcVV1dPSGNGzMCTVDWCIY5Np9XVAPn/AD+dT0sPR+e\nuwluOBKe+l+Ih8bmOzQazYxhsgvF/cDHrNVPxwG9Usq2QjdqzNnf0tiDpXwefOAW+MyzMO8EePxb\nSjBe/AUk49raXKPRjIiC7tYSQtwNnApUCSF2Ad8EnABSyp8DDwHnANuAMHBFYVo6zgSaoGHV+H1+\n7TK49G7Y+QI8ei089GX45/UQ64NUXN1zkNbmqbRkd0+EORW+sW+3RqOZFBRUKKSUl+7nugQ+O0HN\nKQyRAER7xqdHMZjGY+GKh2Dbo3D3pZBODLxuW5sPIRRSSlp7Imxs6WHzrl42tfSwpbWXtIRXrz0L\nhznZO6gajeZg0P4PhSbQrMqJEApQezAWnwnpZP7rOdbmfdEEm1p62NTSw8aWHja29NLZHwPAZRoc\nNruEC1c2sLyhjJSU+i+TRjNN0f+2C81ol8YeLKUNarhpECFPHf/zx028srOHbR39GXeQBdV+Tl5S\nxVFzylgxp4xD60pwOXQPQqOZCWihKDS2UJTNndCvDZ/0DdwPfxEzlXWijUgXX++7gKde38tRc8o4\nd8VsjrSEodTrHObTNBrNdEYLRaEJNIGvEjwl4/o1uwJh1m3vZkNzNxuaA2xtL+FccSVfcd5DvehE\nAHvnnMOXzv8vbqj0IYQY1/ZoNJqpgxaKQjNOS2MDoTjPvdPFM9s6ee6dTpq7wgCUeBwcPbecc5fP\nZuXcYymb8y2Ey4S1pzAv/jZU+tQ8hkaj0VhooSg0gSZlIT5KookUG5oDPL21k2e3dbJldy9SQpHb\nwXELKrj8XfM4flElS2qKMYw8QrDqKrU8duc6mPuuUbdHo9FMH7RQFJJ0Sk0oL7vggN8qpeTtvf08\nvbWDp7Z28sL2LmLJNA5DcHRjOV88fQknLq5kRUPZyJatHnGRsvpY/0stFBqNZgBaKApJX6tapjrC\noadAKM7T2zp56u0Ont7awd4+tVR1YbWfS1c3ctLiKo5bUInffRD/W11+WHEJbPgVrLke/FUH/hka\njWZaooWikOxnaayUki2tfTzyxl6efLuDzbt6kBJKvU5OXFTFyUuqOHFxNfVl3rFpz6or4cVb4JW7\n4MQvjc1najSaKY8WikKSRyiklGza1ctDr7bx0Ktt7ApEMASsmFPGF05fzMlLqlnRUIaZb55htNQc\nCnNPhPW/guO/oHK8NRrNjEcLRSEJNIHhQJbMZuPOAH/d3MbftuyhtSeC0xScuKiKz5++mDMPq6Xc\n75qYNh1zJfzxSnjnMbWDW6PRzHi0UBQIKSU9rVuRzjrO/eHTtPZEcJkGJy2u4t/OXMIZh9VS6ivA\nJrdDzwV/Dbx0qxYKjUYDaKGYUKSUvNEW5K+bd/PXzW3c2P8a/ZRzyIJi/u3MJZy5rJYST4F3QDtc\nKl716f8HPTtVToZGo5nRaKGYAFp7Ivxx/S7u39TKOx0hTENwwqIqDk12Iw47kRMvOKbQTRzIyo/D\nMz+CDbfD6dcUujUajabAaKEYJ9JpydPbOrnr+WYef3MvElg9r4IrTpjPew6vo9IRg+sDUL2g0E3d\nl7JGWHw2vHwnnPI11cvQaDQzFi0UY0xPOM4fN+zi1+uaaeoKU+l38elTFnLp6saB4T573lblRLvG\njpRjroK3H4Y3H4DDLyx0azQaTQEpdMLdGuAGwARulVJeP+j6x4Efks3J/omU8tYJbeQI2dLay53P\nN3Hfxt3EkmlWzi3nS2cuYc3hdbgdebKwC2UvPlIWnq4cbV+6TQuFRjPDKZhQCCFM4GbgTGAX8JIQ\n4n4p5euDbv29lPJzE97AEZBIpfnblj3c8VwT65sDeJ0mHzi6nsuOm8uy2aXDv3myC4VhwKorVHRq\n+5tqj4VGo5mRFLJHsRrYJqXcDiCE+B1wHjBYKCYdwWiCO55r4s7nm2kPxmis8PFf7z2MD66cM/Il\nrYEm8JSBt2xc2zoqjvooPPFdWH8bnPODQrdGo9EUiEIKRT2QG7G2Czg2z30XCiFOBt4GviSl3DeW\nDRBCXA1cDdDYOD5LOqOJFL9e18zNT2wjEE5w8pJqrr9wLqcuqcnvyDoc42QvPqb4q2DpebDpbjjj\nm8oPSqPRzDgm+2T2A8DdUsqYEOJTwB3Au/PdKKVcC6wFWLVqlRzLRiRTaf708i5+/OhW2nqjnLS4\niv84+xCWN4yiNxBogtrDx6yN48aqq+DVP8Crf4SVlxe6NRqNpgAUUihagTk55w1kJ60BkFJ25Zze\nCkzo+IeUkr+/tocf/P0ttneEOHJOGf938QqOXzhKZ9V0Sm1mO/R9Y9PQ8aTxOKhZquzHj/6YDjXS\naGYghRSKl4DFQoj5KIG4BPhw7g1CiFlSyjbr9P3AGxPVuBe2d/G9h99kY0sPi2qKuOWjKzlrae3Y\nRIQG2yAVn/xDT6CEYdWV8NCXofVlaFhZ6BZNLTbfA49dB727oLRBbWBcfnGhW6XRHBAFEwopZVII\n8Tng76jlsbdJKV8TQlwHrJdS3g98XgjxfiAJdAMfH+92tXSHufb+13jszXbqSjz84MLlfODo+pGF\n/4yUyb7iaTDLPwSPfFP1KrRQjJzN96jUwEREnfe2qHPQYqGZUhR0jkJK+RDw0KC6a3KOvw58fYLa\nwp9ebuXa+18D4KtrDuWKE+bhcebZAzFapppQeErUg23T3XDWt8FXUegWTQ0euy4rEjaJiKrXQqGZ\nQujAAdRu6s/+9mW+/IdNLJtdwt++eBKfOXXh+IgEKKEQphqKmCoccxUko7Dxt4VuydShd9eB1Ws0\nk5QZLxTPbetkzY+f5pHX9/LVNYfy208eR0O5b/9vHA2BZiUSZoGdYg+EuiOgYbXaU5FOF7o1U4Mh\nfwhI+N1HoG3zhDZHozlYZqxQxJIpvvPg63z41hfwuU3u/ZcT+MypC8cnOW4wU2EPRT6O+QR0vwM7\nnix0S6YG+fI8HF612m3H03DLSXD3pbD7lYlvm0ZzAMxIodjdE+GCm5/jF0/v4LLjGnnwX0/i8Pr9\nWG6MJYEmKJ87cd83Viw9D7wValJbMzxd78Cm30PlYqtnIaB0Drz/RrjkN/DFzXDqf0Lzs7D2VPjt\nh6B1Q6FbrdHkZbJvuBtz2nojXPqLdXT3x/nl5as4/bDaiW1APASh9qnZo3B64KjL4PmboW83lMwu\ndIsmJ8k4/PEKZc/+sfugtH7fe7xlcOpX4bhPwwtr4fmfwC/eDYvOhFO/Bg2rJr7dGs0QzKgexZ7e\nKJeuVSJx51WrJ14kQM1PwNQUClBGgTKlsio0+Xnsf6BtE5x3c36RyMVTCqf8B3zxVbXHonUD3Ho6\n3PUBaHlxYtqr0eyHGSMUe/uiXPqLdXT2x7njqtUc1VhemIZMtaWxg6lYoCzIN9wBqWShWzP52PqI\n6h0c80k49L0jf5+nBE76dzUkdca10LYRfnkm3HkeND8/Xq3VaEbEjBCK9j7Vk2jvi3LHlcdwdKFE\nAnKEYn7h2jBajrkKgrtVsJEmS3AP3PtpqFkGZ33r4D7DXQwnfkn1MM78Fux9DX61Bu44F5qeHdv2\najQjZNoLRTCa4NJfrGNPX5Q7rlzNyrkF3iwWaAJ3CXgLKFajZfHZUFIPL+lJ7QzpNNz7KTUHddFt\n4PSO7vNcfjjh8/CFzXDWd1QmyO3nwK/eCzueAjmmvpcazbBMe6H4yePbeKcjxK2Xr2LVvEmwo9he\n8TSVzfVMB6z8OGx/Qq3u0cBzN8D2f8J7rh/bkCeXD47/HHxhE6y5Hrq2qd7Fr86Bd55QNiE/Ohyu\nLVPl5nvG7rs1GotpLRTbO/q57dkdfHBlw+gdX8eKqbqHYjBHfwwMh9qAN9PZtR4e/zYsPR+OHicr\ndpcPjvsMfGEjvOcHENgBd52vejG9LYDMeklpsdCMMdNaKL7119fxOEy+smaSxHim09DTPD2EorhO\nTdZu/M2+fkYziWgv/PFKKJ4N594w/j1FpxeO/RR8fqNKSJSDdsnbXlJTBd0jmhJMW6F4/M29PPFW\nB58/fTHVxe5CN0fRv1f5JU0HoQAVahQJwGv3FrolhUFK+OuXlHfThbdObKyt06NEKh9TxUvKdtfV\nPaJJz7QVim//9Q0WVPu5/Ph5hW5Klqm+NHYw809WO49n6qT2xt/Alj/BaV+HxnwpvuPMUF5SU2Uj\n5JDuuv9TmPZohmRaCkU8mWZ7Z4irTpyPyzGJ/ojTYWlsLnaoUet6tcFsJtG5FR76D5h3Epz4b4Vp\nw+nX5F9dZScoTmZCnVZPIg+9u+DZGyDcPbFt0gzJJHqKjh2RRAqAFaPJtB4PAk1kPH+mC0deqozu\nZlKvIhmzLDo88IG1YIyTHf3+WH4xnHuj9ffJ+nt10pchGYFbz5yc7rTptFoAcdMwAVimGx65Bv7f\nUnjgC9BuBVuO9XyGnh8ZMQX1ehJCrAFuQCXc3SqlvH7QdTdwJ7AS6AI+JKVs2t/nRhIpvKZgcW3R\n2Dd6NASa1HCBw1Xolowd3nI44kJ49Q9qk5lnAs0VC8Uj34Q9r8Klvy/8MM/yi/cNQTriIvj1hWoJ\n7SW/hgWnFqJl+7L7FXjw35VNybyTYNEZ8OT1A4efnF4lfjVL4YWfw6bfwYbboepQCGxXEcJw4GmB\ngyNpF58Fm347MH3wL/8CD39VzbvlxtZm3tuicmRkChBAzl4WbwW85/vTNpCqYD0KIYQJ3Ay8B1gK\nXCqEWDrotquAgJRyEfAj4Psj+exIPMWS2mLcjgL90huK6bI0djCrroJEWLmlTnfe+hu88DM49tNw\nyJpCtyY/NYfBVY9A2Rz49UWF/6UcCSiBWHsa9LTAB26Fyx+AE7+4b4/o3BvVw7bucDjvJ/Cl1+H0\nb0LX1qxI2Ix0hVe+SfP1v9x3fiSdgEh39p6//At8Zzb8+ZPZYTKZsm4etOEx0g33fbbw/63HiUIO\nPa0Gtkkpt0sp48DvgPMG3XMecId1/EfgdCH2v/4wmkixdFbJmDZ2TJiq9uL7o/5omH2U+sc3nXcM\n97XBXz6jQpzOnORLUEvr4YqHofE49aB79oaJ/38jJWy8G25apYabjv0U/Ot6WP7B7DLi5RfDl7bA\ntT2qHPyL3F8JJ/3bvsuAbUaywivfpPlISCcgERr5/an41FqafAAUcuipHsidzdoFDF46krlHSpkU\nQvQClUDn4A8TQlwNXA3grVuYmaeYNMTD0L9nevYoQPUq7v8cND8H804odGvGnnRKPXCTUbjoV+CY\nJEuuh8NbBpf9SW3Ke+QaZQ1/9nfHb04ld3inqBZcxdC9FRqOgff+GWatOPjPLm0YYvJbwq1nKBNG\nmYInvpsdXjrxS0oghpo0Hw+mytLkA2Ta5FFIKdcCawGq5y+Vr+3uK3CLBmGvQpkuK54Gc/iF8I9v\nqF7FdBSKZ34ETU8r6/CqxYVuzchxuOHC29SGwHU3Q7ANLlir9mGMJfbwjv3LvX8PsEft4H/fDWCM\ncvDi9GsGfj6oRRRL36/mPO69euD9vS3woLUazXTtO2wF7DPPMBYMGX87tSnk0FMrkLv8p8Gqy3uP\nEMIBlKImtYfF6zLZ0RkiGE2MUVPHgJ4pnkOxP1w+WPFheP1+6G8vdGvGlpYX1S/Vwy+EIz9S6NYc\nOIYBa76rzAVfvw/uukDNG4wlj3wz//DOO0+MXiQg/wqv99+oVp199iXwV+d/X1GdEvfBy4idXrW0\n2/48b4USlNFgupSgTUMK2aN4CVgshJiPEoRLgA8Puud+4HLgeeAi4HEp9z/Q6nWaxIDXd/dx7ILK\nsW31wTLdNtvlY9WVaqL3lbtUtsJ0INIDf7xK/VJ834+mtpnj8Z9T1it/+QzctkYNS432F3DXO/Dc\nTcp2Ph9jORSTb4UXKCEK7TMarejfm31P7qone0VTLrlDZ95yiPfv2xMRhjVfMrNWPRVMKKw5h88B\nf0ctj71NSvmaEOI6YL2U8n7gl8BdQohtQDdKTPaL16WEYn1zYHIJhasIfJOkPeNB9RK17HH97XDC\nFwu3v2CskFKt4w/uhiv/Pj2W/h5xERTVwO8+ovZaXPZHqF124J+zeyM8+2PVQzGcyhY9nmfid6KG\nYoaaw7C/fyiRyWXwPYOX1OYTlxlCQTfcSSkfklIukVIulFJ+x6q7xhIJpJRRKeUHpZSLpJSrpZTb\nR/K5DkNwwqJK1j61nUAo39hkAQg0QdkUtxcfCcdcBb07YdujhW7J6Hn5Tnj9L/Du/5peGdbzT1Yr\nopCqZ7Hj6ZG9T0plpX7nebD2FNj2GJzwBRWy9L4f5x/emaihmHy71Ef7/ftbkTWDmJY7swGued8y\n+mNJ/u+RtwrdFMV03UMxmEPfp1a8vHRroVsyOjreUpuvFpwKx3+h0K0Ze+oOV3stSmbDrz+gPKuG\nIp1Sxo9rT1Ui0f4GnPE/6uF5xrVQXJt/DsHeEzERFPr7pznTZtXTYA6pK+ajx83lzueb+PDquSyd\nXcB9FVIqoVhwWuHaMFGYTrXS5an/nbrimIgq63CXHy64ZWwmYycjZXNUz+J3H1F/3rcehp3rskMt\np34dUjE1B9G9HSoWqofvikvyLw8eyfDOeFLo75/GTNN/AYovnbGEUq+Tax94jRHMgY8foQ61c3kq\nPjQPhpUfV0NsG24vdEsOjkf+G/ZugQt+riZ/pzO+CvjovTD7aGXDkrt7+b5/UTbqnjK4+E743Euw\n8vKpsYdEM6ZMa6Eo9Tn58tmH8OKObn75zI7CNWQmrHjKpbQBlqyBl+9SBnpTiTcfhBfXwrs+B4vP\nLHRrJganB0JDLGn2V8MnH4el5039xQmag2ZaCwXAJcc0cvayWr794Bvc+XxTYRox04QC1E7tcCe8\n8UChWzJyeluVX8+sFdN2PfyQ9A7ewmQR6pz+CzA0+2XaC4VpCG669GjOXFrLNfe9xl3PN018I2yh\nKGuc+O8uFAvfrYRxqtiPp1Pw56shGZ86Fh1jyVDLWKfpTmPNgTHthQLA5TC4+cNHc8ZhNfz3fa/x\n63XNE9uAQJOyUBhr24TJjGGoDXg7n4O9rxe6Nfvn6f+D5mfgvf8HlQsL3ZqJZzyWl2qmDTNCKMAS\ni48czemH1vBff9nCb1+YwASwqbr6Z7QceZkKoVl/W6FbMjzNz8M/vwfLP6SCmGYienmpZhim7fLY\nfLgdJj+97Gg+fdcG/vPeV2nuDvHvZx4y/nGpgSaYf8r4fsdkxF8Jy85X4TNnXAvuSRYkBcrz6E+f\nUJshz/nfQremsOjlpZohmDE9Chu3w+TnH13JpasbueXJ7Vzw02fZ1h4cvy9MRJW980zsUYCa1I4H\n1dLLyUJuBOb/HQZ9rXDRL8EzCTNMNJpJwIwTClBi8b0PHMEtH13J7p4I77vpGe56vml89lrY69Jn\nqlDMWQ21h0+eUKPBaWfJCBgOZW6n0WjyMiOFwubsZXX8/Ysns3p+Jf9932tcdcd6OoJjvO5/Ji6N\nzUUINam951XYtb7QrcmfdpZOTNtkMo1mLJjRQgFQU+Lh9o8fwzfPXcoz2zpZ8+OneOT1vWP3BTNd\nKECNe7uKVK+i0Axlez1Nk8k0mrFgxgsFgGEIrjhhPg987kRqSjx88s71XHX7S7y1ZwzmLgJNKomr\nqGb0nzVVcRerFUVb/gzh7sK0oW0z/PZDDJlopvcLaDRDooUih0Pqirnvsyfw1TWH8mJTN2tueIp/\nv2cTrT0HEcxuYy+Nnem7W4+5ShnMvfLrif3ejrfgnsvhlpNg5/Ow9Hwl3Lno/QIazbAURCiEEBVC\niEeEEFutsnyI+1JCiI3W6/6JaJvLYfCZUxfy9FdO45MnLeCBzbs57X//ybf++jpd/QcxfzFT91AM\npnYZzDlO7alIp8f/+7p3wL2fhp8ep7IxTv4P+MJmuPgOFaGp9wtoNCNGFMJVVQjxA6BbSnm9EOJr\nQLmU8qt57uuXUh7w4vtVq1bJ9evHZuK0tSfCjx95mz+9vAuv0+STJy/gEyctoMg9gi0oUsL3GuCo\nj8J7rh+T9kxpNt8Df/4kXPZnWHT6+HxHbys89UMVx2o4YPUnVdqev2p8vk+jmSYIITZIKfMmdBVq\n6Ok84A7r+A7g/AK1Y7/Ul3n54QdX8I8vnczJS6r58aNbOfkHT3Dr09uJJlLDvzncrXJ3dY9CsfQ8\nFQU7Hju1+9vhb1+HG49Sw1srr4DPb4Szvq1FQqMZJYXamV0rpWyzjvcAtUPc5xFCrAeSwPVSyr9M\nSOvysKimmJ9dtpJNLT388O9v8e0H3+DmJ7bxoWMauey4RhrKffu+Sa94GojDDUddpoJweluhtH70\nnxnuhuduhBduUZbmR14KJ38FyueO/rM1Gg0wjkIhhHgUyJf68o3cEymlFEIMNf41V0rZKoRYADwu\nhHhVSpl3Z5QQ4mrgaoDGxvFzaV0xp4xff+JY1m3v4vZnm1j71Dusfeodzjislo8fP493LaxE2BPX\nASsDQwtFlpVXwLM3wst3wGn/efCfE+2DdT+D538CsSAcfqFKZKtaNHZt1Wg0wDgKhZTyjKGuCSH2\nCiFmSSnbhBCzgLypKVLKVqvcLoT4J3AUkFcopJRrgbWg5ihG2fz9ctyCSo5bUElrT4TfrGvmdy+1\n8I/X97KopoiPHjeXDxxdT/FMtBffHxXzYdEZsOEONcFsOg/s/fEwvPQLeObHEOlWGd2n/aeaLNdo\nNONCoeYo7gcut44vB+4bfIMQolwI4baOq4ATgEnnV11f5uUraw7lua+9mx9etBy/y+Sb97/Gsd99\njPUbXyHpqwFXnmGpmcwxV0H/HnjroZG/JxmDF9bCjUfCI9dA/dHwySfgkt9okdBoxplCzVFcD9wj\nhLgKaAYuBhBCrAI+LaX8BHAYcIsQIo0StOullJNOKGw8TpMPrprDB1fNYVNLD3etayb56g42Usb3\nf/4cHzl2Lmcvq8Pr0nGSLD5LLUt96Zdqgns4UgnYdDc8+QPlzzT3BPjg7TD3+AlpqkajKZBQSCm7\ngH3WR0op1wOfsI6fA46Y4KaNCSvmlLFiThmplj62e4+gPRjji7/fiM9lcsZhtbx/xWxOWlKF2zFD\nRcMwYeXl8Pi3oXNb/nmFdAq2/EnlRHRvh/qV8P6bYMGpevOiRjPBzKg8igklGccMtrL4qA/zxCmn\nsm5HFw9sauPhLW3cv2k3JR4HZy2r433LZ3HCoiqc5gzbJH/Ux+Cf16ulsmu+m62XUuVsP/Fd6HhD\nOc9e+jtYskYLhEZTILRQjBe9LSDTUD4PwxAcv7CK4xdWcd15y3hmWycPbNzN37fs4Y8bdlHmc3L2\n0jreu3wWxy+sxDETRKO4FmYdCet+ql6lDbDsAtjxJLRtgsrFKrt66fkqVlWj0RQMLRTjxRB7KJym\nwWmH1HDaITXEkimefruTB19t48FX2/j9+hbKfU7OWlrHOZZoTNuexuZ7lPW4bdLX26L2Q3gr4Pyf\nwREXg6n/emo0kwH9L3G8GMFmO7fD5IyltZyxtJZoIsWTb3fwUI5olHqdnLW0lrOX1fGuhZX4R2Ib\nMlV47DplEjgYlw+O/PDEt0ej0QzJNHryTDICTWC6oSjfnsN98ThNzl5Wx9nL6ogmUjy9tZOHX23j\nb1v28IcNu3CagmPmVXDKkmpOOaSaQ2qLsxv7piJD5kK0Tmw7NBrNftFCMV4EmpSNxEGMr3ucJmcu\nreXMpbXEkinWNwV48u0Onnyrg+89/Cbfe/hNaordnLS4mpOXVHHCoiqqitxj/2cYT0obrDjSPPUa\njWZSMaxQCCFKgOrBthlCiOVSys3j2rKpzhjZi7sdJicsUmLwn+ccRltvhKff7uSprR089uZe/vSy\n+mW+dFYJJy2p4qRF1ayaV7zBol4AACAASURBVI7HOcmX3p5+jcquzo0l1bkQGs2kZEihEEJcDPwY\naBdCOIGPSylfsi7fDhw9/s2bokiphKLxuDH/6FmlXi4+Zg4XHzOHVFry2u5ent7ayVNvd3DbMzu4\n5cntuBwGq+dVcMKiKk5aXMXSWSUYxiQbprLzHx67Tg1DlTYokdC5EBrNpGPIPAohxEbgPZYf02rg\nTuDrUsp7hRCvSCmPmsiGHghjmUdxUIS74Qfz4ezvwrs+O2FfG4oleXFHN89s6+SZrZ28tVdFuZb5\nnLxrQSXvWljJqrkVHFJXjDnZhEOj0RSU4fIohht6Mm0rcCnli0KI04C/CiHmMGTwsAYomL243+3g\ntENrOO1Qlc/dHozy3LYunt3WyXPvdPHwlj0AFLsdHNlYxtGN5aycW86RjWWUeA7QnE+j0cwYhhOK\noBBioT0/YfUsTgX+AmgXtuGYJDkUNcUezj+qnvOPqkdKya5AhPXN3axvCrChOcCNj29FSrXheUlN\nMUfPLeOoOeUc1VjGwuqiyTdcpdFoCsJwQvEZwBBCLLXN+KSUQSHEGuCSCWndVCVjLz55wnOEEMyp\n8DGnwscFR6mVRcFogk0tvby8UwnHg5vbuPtFtRLJ7nUcZflWHTmnjMqptrJKo9GMCUMKhZRyE4AQ\nYosQ4i7gB4DHKlcBd01IC6cigSbwV4P7gOO+J5Rij5MTF1dx4mIVFZpOS3Z0hXhlZw+v7Azwys4e\nfvLENtLWQOOcCi8rGpRorJhTxrLZJfhceoW1RjPdGcm/8mOB7wPPAcXAb1DZEJqh6Gku+LDTwWAY\ngoXVRSysLuKilarXEY4n2dLax8aWABtbenhlZw9/3axSbA0BS2qLWd5QyhENZSydVcwhdSUUTacd\n5BqNZkRCkQAigBfVo9ghpUyPa6umOoEmaDim0K0YE3wuB6vnV7B6fkWmrj0YZXNLL5t39bBpVy//\neH0v96zP7rRurPBxaF0xh80q4dC6Yg6pK2ZupV+vtNJopigjEYqXUAl0xwBVwM+FEBdKKT84ri2b\nqqSS0NMCR0zf/zw1xR7OWOrhjKW1AEgp2d0b5c22Pt5o6+ONPUHeaOvj0Tf2ZoatPE6DJbXFHFKr\nhOOQOnVcXeye2lYkGs0MYCRCcZUVKATQBpwnhPjoaL5UCPFB4FpUit3qnM8ffN8a4AbABG6VUl4/\nmu+dEPp2gUxNyaGng0UIQX2Zl/oyL6cfVpupj8RTbG0P8uaeIG9Zryfe6uAPG7K9j1Kvk0Nqi1lS\nV8TimmIW16qyqsilBUSjmSTsVyjyPcSllKOdyN4CfAC4ZagbhBAmcDNwJrALeEkIcf9kjkMFJuWK\np0LhdZksbyhjeUPZgPqu/hhv7+3n7b1B3tobZOveIPdv3E1fNJm5p9znZHFNMYtqi1hUXcSimiIW\n1xZRV+LRAqLRTDCFikJ9A9jfP/jVwDYp5Xbr3t8B5wFTQyhmUI/iQKkscvOuIjfvWliZqZNS0h6M\nsdUSkK3t/WzdG+TBzW30RhKZ+4rcDhZW+1lYXcT8Kj9zq/zMq/Qxt9JPqVdvGtRoxoPJvDylHsi1\nF92FWoGVFyHE1cDVAI2NjePbsuEINIHhhJLZhWvDFEQIQW2Jh9oST2a5LigB6eyPs629n20d/Wzb\nG2R7Z4jnt3fx51cGWpKX+ZzMrfDRWOm3Sh+NFepVV+LRGwg1moNk3IRCCPEokC+M4RtSyvvG+vuk\nlGuBtaC8nsb680dMoAnKGsGY5O6tUwQhBNXFbqqLB/ZAQM2B7OwO09QVorkrRHNXmJ3dYTa19PDQ\nq22k0tm/Bi7ToKHca2069DKn3EdDefa4zOfUQ1oazRCMm1BIKc8Y5Ue0AnNyzhususnNGNmLa/aP\n12VmVlANJpFKs7snws7uMC3ddmkJya4eesKJAff7XSb15V4ayn00lKuJ+Xq7LPNSVeTWPRLNjGUy\nDz29BCwWQsxHCcQlwOTPyAw0wWztwF5onKbB3Eo/cyv9ea8HowlauiO0BJSAtPZE2BWI0BqIsL6p\ne8DEOqgeyawyD7NLvcwq81Bf5mWWdWzXFbsduleimZYURCiEEBcANwHVwINCiI1SyrOFELNRy2DP\nkVImhRCfA/6OWh57m5TytUK0d8REeiAS0D2KKUCxx8nS2U6Wzi7Je70vmmB3T4TdPUo8WnuitFrn\n697pYk9flPSgAU6/y6Su1MOsUi91pR5qit3UFLupLvZQU2Ifu7XtiWbKUahVT/cC9+ap3w2ck3P+\nEPDQBDZtdPQ0q1ILxZSnxOOkpM7JoXX5hSSZStMejNHWq0RkT2+Ett4oe3qjtPVGeWZrJ539MZKD\n1QQlKNXFbqqK3JmyqshNZZHLOnZRaZ3rXopmMqB/2owlemnsjMFhGswu8zK7zMvKIbbMpNOSQDhO\nezBGRzCWKTv7VdkRjLGtvZ/nt3ftM2di4zINKvwuKotcqvS7KB9c+tRxmc9Juc+F0zzwnHaNZji0\nUIwlGaHQm+00ymRR9QzcHDZr+HsTqTTdoTgdwRhdoTidwRjdoTidoRjd/XG6QnG6+mM0dYUIhBL0\nx5JDflax20GZ30mZV4lHmc9Fuc9JmddJiddJqVfVqdJJiUfVeZyG7r1o8qKFYiwJNIG3AjylhW6J\nZorhNI3MPpKREE2k6Akn6ArFCIQSBMJxesJxAuEE3SF13BNJ0BNO0NIdpieSoDeSYIjkY6sNQg25\neZ2UeBwUe5wUexzWSx0XubPnRW4Hfuvc73ZQ5HLgd5s4dI9m2qGFYizRS2M1E4THaVJXqibPR0o6\nLQlGk/RaotETidMbSdAXSdIXTVjHqgxGkwSjCfb2RemLqvNwPDWi73E7DIrcDnxuE7/Lgc9l4nc7\n8DpNvC4Tr9PEM+DYwOM08ThM3Pax08TtMKyXqnc7DFwOA7dp4rKOtSPx8EgpSaYlsWSaeM4rlkwR\ns8tEmlhyeEPwaSkUrT0Rfv/STpY3lLG4pmjifuEEmmDWkRPzXRrNAWIYglKfk1LfwVmdJFNpQrEU\nwZga+uqPJgnGkoSsV38slTkOxZOEYyn6Y0pggtEkHcEY0USKcDxFJJEiEk/lnew/EExD4DINnKZQ\n4mEaOEwDhylwGgZOh8BhqOumIXCaSlwchqo3DZF5GULVGwYYQlgv9d9NoI6FUNcQIBDYI3WD5Sr3\nTyUlSKQqpSQtVV1aypyXEvJUWpKSknRaPeDTUpJMqfpEWpJKp0lY58lUmnhKlcm0JJFKWy+pBME6\nH64XOVKmpVD0hBN89U+vAsreeumsEpY3lHFEfSnLG0pZUF009r9E0ino2QlLzx/bz9VoJgkO06DU\nZxy00OQjmUoTTaaJJlI5L/ULN5bI/uqNJqxfwqmBv4oHPxjjyTTJVJqE9SBNpNQD1H7YhpJJkmn1\n8E2m06TS6iGdTKdJp60y54GeSquHtiTnIY8qbTWQObJgZ9CDEhIbkSMyAuU4YIuQKUTmPCNYpqq3\nRcxhCkzDwGmde5wGDrcDpyWSDlNdc5qqp+U0lUi6TCWebqcqXY5sb8zjMLI9N6fJUd8f5v/9mP0f\nn0Qsm13CPf9+Cq+29rLJCtj5/Ust3P5cEwA+l8my2SUcXl/KEdZr1OLR1wrppB560mgOAIdpUGQa\nOhVxkjNt/+8sqC5iQXUR5x1ZD6hfBu909PPqrl5ebVXicfeLO/lVQo3NeZ0mS2eXcER9KUtnl3D4\n7FIW1xaNfKmhXhqr0WimKdNWKAZjGoIltcUsqS3mQisPOplKs70zlBGP13b3cs/6lsykncthcGhd\nsRXnWZI5rixy7/sFAb3ZTqPRTE9mjFDkw2Ea+4hHKi1p6gqxpbWX13b3saW1l8feaB+QCV1V5M5k\nQat0tmKWdm7HZTigpL5QfxyNRqMZF2a0UOTDNAQLq4tYmDNsBdARjPHWniBv7ulTsZ57g/zmhWai\n1tDVDc51rHRUce2vX2GxlQ29uFZ9jsepLcc1Gs3URQvFCLEzEXJDdVJpSUt3mLf2Blnxtz4CqQaa\nu8L8862OzLI/Q8C8Sn8mC9ouF1T7tYBoNJopgRaKUWAagnlVfuZV+eHBPdQddi6PnHsK8WSapq4Q\nb+1RedBv7+1na3uQR99oz4TpGAIaK3wsqiliUU2xyoSuKWJhTZFeAaLRaCYV+ok0FsSCEO7MTGS7\nHNm5jwG3JVM0dYYzmdDvtCsBefLtDhKp7FrsuhIPC2v8zK/yM7+qiPlVPuZV+mko9+FyaHsEjUYz\nsWihGAsyK56GNwN0O/InsiVTaZq7wyoX2hKQ7Z0h7t+4e0CAjmkI6su8zKvyM7/Sp3ozlX7mVvqY\nU+HTrqEajWZc0EIxFoxyD4XDNDIT6Gcvy9ZLKekOxWnqCtHUqbKhd3SGaOoK8XJzYICDqC0icyt9\nNFb4rNLPvCp1rsNyNBrNwVKohLsPAtcChwGrpZTrh7ivCQgCKSAppVw1UW08IMZps50QWZvqlXMr\nBlyTUtIVitOcIyJNXWF2doX46+Y2eiMD8w2qitwZEcm8rPNqnQet0WiGoVA/M7cAHwBuGcG9p0kp\nO8e5PaMj0KSsxb3lE/aVQohMMtpgEQHoDSdo7lbi0dIdprkrxM7uMC9s7+IvG1sHGIW5HAZzyr3M\nqfAxp9zHnAovc8p9NFjHpV6nzinQaGYwhYpCfQOYPg+fSWgvXupzstxXxvKGsn2uxZIpWgMRdnaH\naQlEaOkOs7MrTEsgzMvNgQHzIqCCcOrLvVaim8qEnl3mYXapqqst8ehJdo1mGjPZB64l8A8hhARu\nkVKuHepGIcTVwNUAjY2NE9Q8i0AT1C6d2O8cBW6HmfHCykdvRIXd7AqEaemO0NqjXrt7IryyM0Bg\nUGynEGpoa3aph7pSJSSzSj3MKlNlXYmHmhI3bofeN6LRTEXGTSiEEI8CdXkufUNKed8IP+ZEKWWr\nEKIGeEQI8aaU8ql8N1oishZg1apVY+DAPkLSaehphkPPmbCvHG9KvU5K60s5vD5/Ul84nqStN8ru\nnghtPVF29yoRaeuNsr0jxHPbugjmieqs9Luos4Sj1ipzj2tL3HqYS6OZhIybUEgpzxiDz2i1ynYh\nxL3AaiCvUBSMYBuk4pNu6Gk88bkcmVVaQxGMJtjTG6WtN8qevmjmeG9flN29UV5p6aE7FN/nfS6H\nQW2Jm9piFQtaXeymtsRDTbGbmhI3NcXquMynBUWjmSgm7dCTEMIPGFLKoHV8FnBdgZu1L9pePC8q\nY9nJ4kGbDnOJJVO098UyQtIejNHep8Rkb1+MN9r6ePLt2IBlwDZOU1Bd5Ka6xEN1kRKR6iJls1Jl\nlTXFbiqLXHppsEYzSgq1PPYC4CagGnhQCLFRSnm2EGI2cKuU8hygFrjX+tXoAH4rpfxbIdo7LFoo\nDhq3w1QrrSp8w94Xjidp74uxt88Sk2CMjmCM9mCUjmCMXYEwL+8MEAjH88Y++lwmlUUuKvxuqvwu\nKvwuKopcVPpdlPtcmWsVPhflfidFbofurWg0ORRq1dO9wL156ncD51jH24EVE9y0AyfQBMKA0jmF\nbsm0xedyMK/KoTy1hiGRStMditMRjNHRr8Sksz9GV3+c7lCczn7Ve3m9rY+uUJz4EIHyTlNQ7lOC\nUuZzUu5zUe53UW4dl/nUcZnPaR27KPE4Ji6bXaOZYHSffLQEmqC0AcyxyxHWHBxO06C2RM1t7A8p\nJaF4iu7+OF2hGIFwnO5QgkAoTnc4Tne/KnvCcba29xMIxemJJDKmjvko9jgo8zkp9Top87oo9Top\n8TozdbmvEo+TEq+DUq8aohvzDHeNZgzRQjFaJuEeCs3+EUJQ5HZQ5HbQWDn80JdNOi0JxpL0hOP0\nhBMEcsreSIKecILeiHoFwnF290bos85zTR/zUeR2UOJxUGwJiJrjcVBilcWZUr2K3GqITB078Lsd\nei+LZtzQQjFaAk1wyJpCt0IzARiGyPQI5laO/H1SSiKJFD3hBH3RBL3hBH3RZEZEeiMJgtEkfdEE\nfdbxnt4o29qTBKPqPDlMT8bG5TAotkTD73ZQ5Dazxy4HPrdJkduBz+XA7zZV6TLxutSxz2VaL4dV\nZ2qjSQ2ghWJ0xEMQatc9Cs2wCCGsB7GD2XgP+P1SSqKJtBKNWJJgNEl/NEl/TAlOKGafZ18hq+zq\nj7OzK0woniQUSxGKJ/NO+A+FwxB4nSYel4nHaeBxKGHxOKw6h4HbqUqP08SdU7qdBm6Hicth4HYY\nVqnOXaa67jKNzLnTKtWxwGkaOAyhFxaMI+m0JJFO77fHOy2Foq03ym9eaGZ+pZ/51X5qiz3jY3rX\ns1OVWig044gQAq/1y79mlJ9li04oniRsCUc4niJslZG4qotYx9Fkikg8TSSRIhJPEk2kiSZTRBMp\neiMJ2hMpYsk00YSqs49H0AEaMS7TwGEKHIbA5TBwGOrcFhJHplT3mIbAYRiY1rFpCEwhME2rNASG\nEJiGcl0WQtUbQv23NjLHYAh1XQgQYJXZc0YiYlIiVYFEIiWkc46llKQlpKV9TZKWklRaXUulJSkp\nSaclybR9Lfuy65IpdZxMS5KpNKm0JJFKW+fqOJFKq2NLHIabc8tlWgpFV3+Mb9y7JXPudZrMrfQx\nv8rP3Ep/JghofpWf6mL3wf9i0UtjNVOMXNFh6P2SoyaZShNL2q8UsUSaeCptlanMtURS1SdSaeLJ\nNPGUJJ5Mk7Tr7AdcUj3w4il1TT3sJKmcB579EExZQ33JdPbhmkqnsw9XKUmnVZSx/VC2H9Qp9QTP\n1EnUvZmHOurhbT/4R0pWaCwRssXGEiNDCARqeNPIESjTyF53WEJnGFnBc5i2MKp7vE4T0xA4zaxg\nKgE1cNriapXOTKmOP/X9ods/LYXi8PpS7v/au9nRGWJ7Z4gdHSrD4a29QR59Y++AbpbPZTK30s+8\nSl+mbKxUQlJXsp+eSEYo5o/vH0ijmWI4TAOHaeB3F7olmpHyqWGuTUuhACynUy8nLKoaUJ9Mpdnd\nE2VHVyiT5dDcFeLtvUEee6OdeCq7tt7lMFQIkJXdMLdCiUljpY/6Mi+eQBO4SybUXlyj0Wgmmmkr\nFEPhMA0V2FPpQ20Mz5JKS3b3KPvtpq4QO7tU2dwV5vntXYTjqcy9QsCdnpeoN6u44fcbrfwGLw3l\nPsuS26PdUjUazbRgxgnFcJiGyFhKDO6JSCnp7I+zs1sJR0t3hMUvddEs6nl5Z4C/bm7bZ2KotsRN\nfVlWPOrLvNSXe2mwSu1BpNFopgL6STVChBBUFyuzuZVzK9RM1ro91B3zPp4++90kU2naeqMquyEQ\nYVcgwq5AmF2BCBtbenh4S9s+S9DKfE5ml3ozIjK7zJMZMqsv8+qIUo1GMynQQnGw9O+FZDSz4slh\nGsMa3KXSko5gjNYeJR52ENDunig7u8I8/07XPi6pDkNQV6qS5GblJMtlgoFKPVT4XXqduUajGVe0\nUBwsB7jiybQe+nWlHlbOzX9PbyRBmxUC1NoTpc0Wk94oL+8MsKd3316Jy2FkRGNWqTcTDJRbVvpd\n2rBOo9EcNFooDpZx2ENh20McWleS93o6LekMxWjridLWqxLl9vSqIKA9vRFe3NFNezC6j5gYVlRp\njRUIVFPiptoKAKqxg4FKVI6DtmzQaDSD0UJxsASaAAFlE2cvbhjCSnjzsGJOWd570mlJVyjO3r5s\nulyHFQTUHlR1m3b10BXKn91Q4XdRY83FZF5F2eMaKxhIR5ZqNDOHQgUX/RA4F4gD7wBXSCl78ty3\nBrgBMFGBRtdPaEOHYvM98OwNgISbVsLp18DyiwvdKkCJif1QHyrzGlR2Q1d/PCcMKGqFAakch45g\njO0dITqCsQF7S2xcpkFlkYuqIjdVdlnsptLvyqTM2dfLfS5to63RTGEK1aN4BPi6lDIphPg+8HXg\nq7k3CCFM4GbgTGAX8JIQ4n4p5esT3tpcNt8DD3weEhF13tuizmHSiMVIcJpGZs5kOKSU9EWSmSCg\njn4VV9rZr4KAOvuVuLze1kdXfzyvy6kQUGElyVX63VQUuaykObdVp0KC7KS5Uq/OZ9BoJhOFSrj7\nR87pOuCiPLetBrZZSXcIIX4HnAcUVigeuy4rEjaJiKqfQkIxUoQQlPqclPqcLKoZ3hwonZb0RROW\ngMQz6XJd/TE6Q6rs6o/z+u4+uvpj9EX3zcIGNadSZiXM2fGkFVZsaW6ZTZhTQUB6KbFGMz5MhjmK\nK4Hf56mvB1pyzncBxw71IUKIq4GrARobG8eyfQPp3XVg9TMIwxCUWVGhi0Zgc5pIpQmE4nSF4iqu\nNBynuz9Gt1XXbb12dIbY0NxDIBwf0u1SCCjxWPGkXielPpcqcxLmSrzOTF1JTul3mXq+RaMZhnET\nCiHEo0BdnkvfkFLeZ93zDSAJ/Ga03yelXAusBVi1atUYmhwPorRBDTflq9ccEE7ToKbEQ80Ioksh\nOwwWCCtR6c1JmeuxU+aspLmeSILmrhC9ERUGNJybssMQlHidlHgcVmmLiJU4Z9VnE+ey6XIlHidF\nHoceKtNMa8ZNKKSUZwx3XQjxceB9wOlS5jXsbQVylxQ1WHWF5fRrBs5RADi9ql4zruQOg83DP+L3\npdOS/niS3pyo0sGvYDRBXySbMrenL5qpjyb2ncwfjN9lUmRFluZGlBa5HareKv3ubL0/p/Rb6XNe\np+7daCYfhVr1tAb4CnCKlDI8xG0vAYuFEPNRAnEJ8OEJauLQ2PMQj12nhptKGybVqifNvhiGoMSj\negoHs5g5nlTpcnZ8aTCajSjts8p+K2UuGLOvJ2nrjWbT50aYLCcE+K2oUr8VX2pHlvrcDnxOFW/q\ndZlWjGk2wtTrHBhj6nWamewJr1PHmmoOHpH/x/w4f6kQ2wA30GVVrZNSfloIMRu1DPYc675zgB+j\nlsfeJqX8zkg+f9WqVXL9+vXj0HKN5uBIpyXhRIqQFWUayokrDcWT9MdSmbqQfWylzoViA0s7jW4k\nOdq55Maaep0q2tTrNFWUqdPE61Qxph6HumZHnLoz0aZWaUWa2vGmuVGnudGmLocxIBhH95QmN0KI\nDVLKVfmuFWrV06Ih6ncD5+ScPwQ8dKCf3xtJ8MrOAPVlXqq0sZ5mEmAYIjPkVJt/4/0BE0+mM6Jh\nx5iG40krtjSlSvs4niJsxZWqVzpzTybWtC97LWql0kWTqQNKchsOl52qZglIbsSpLSgOU+C0U9lM\nA2cm4tQYkOZmWolthrDOrfQ3h5FNgDNyYlBVnZ0gl41CzY0+NezI00EpdOrxoZ4huUl12dp9E1Fz\n/5vZqXjZY3VkJ+bZEaj2fWkrgS83GjVlp/DlRKHacamDo1FT1nkyZUWkpu1Y1Gx0ajITkZqNSh2O\nybDqaczZ2R3mgp8+B6i/nLPKPJY7q+3MqlxabZM9bfetmYqoX/MuyvL7UI4JUqr4UTsLO57c9zgT\nY2rFmubW5V5LpCWJZDbiNJ5Mk0ynB+Y5p2Xm/lA8NSD7OfuQs7OhVV027lQecC9rumEIBuSFG0It\nGskIrCXE5qCscacx/LBkQYaexpvDVxwlf3z339jda1l+90Qsg70oe4PRfX4hlfuclmhk3VlzXVrr\nSj16fFejmSKkc35VZ359p3N/mVuZ2VKq/OucjGy1/Nquy2ZmQzY328Y+Vgna2Tq7dyHIdjNyexwi\nT2a2YGB+tt27MQx1zTSyPR/TsPO07d4SmQzt0QzvTbqhp/HG4zQ5Y2lt3muJVJo9Vm6EcmqNsrtH\nGeztCoR5cUfXPhvBhGWqN9sSjVwBqStR5zUlbjxOnWin0RQawxAYCPQ/x7FjWgrFcDj3kxsBEIol\nMyKi3FkjyrG1L8r2jhDPbesiGNt3V3GF35Wx9q4tsW2+3dSUeDKurRU+l54z0Wg0U4oZJxQjwe92\nsKimmEU1xUPeE4wmMg6tbb1R9lp233v7lLhsalEOrYNxGEK5s5Zkbb5rbOtvywrcNtXTw10ajWYy\noIXiICm2dugOJyaxZIp2y967vS+WcWq1Lb9busNsaA7QnUdQQPVQci2+cy2/qzKl8j7SvRSNRjNe\naKEYR9wOc7/DXKCWOdourB2DLL/b+5RD647OEB39MeLJfXcJm4ag0p+1+q7yu1Rp2XxX5liBV/hd\nuqei0WgOCC0UkwCXw8gs3R0OKSV90aQSFUtA7FdHMOvY+k57/5CiAlDmc1LpV5bflUVZe+/KjNW3\nZQfuV86sOkZVo5nZaKGYQgghMnGpC6uHt/yWUhKMJekMKjdW2/q7yxITu25rez8v7IgTCOdPvBNC\nRbQq8VDDXJVFuVbfSkzsstznokTnSWg00wotFNMUIbL+Rguq939/MpUmEE5YFt8qN8K2+w6E4pn6\n5q4wL+/cv+V3qWXprWzH8x+XepXBn239XerVvReNZjKihUIDgMM0MhPmMPQEvY09DNYTjhMIJwiE\n4vRE4gRCyvK7J5IgYNl/d/XHeaejn56wMswbjiK3IycvwpERkFLb/ts30Aa8xJPNlnA7DO0npNGM\nA1ooNAdF7jDY3MqRvy+RStNnWXv3RBIZ62+VJ5G06uOZe3Z0huiz6iOJ1LCf7TRFRjiy2REO6+Uc\nWLotS3D7unXucWqx0WgGo4VCM6E4TYNKayXWgRJPpjN5EX3RZCaUqC9qZ0pkbcDt+/b2RTO24KH4\n8EIDagWZbd5XnJsf4XFQ5LLOPQ6K3GY2T8KVvc9n5Ur4LUtwvWxZMx3QQqGZMrgchloCfBAiA8rH\np39QhoSdKxG0cyNiCStXImsHHgjHaQmE6bftwUcgODZep4nfypTwuZS42PkRuXUe58AMicxxji24\nfc1jWYS7TN370UwMWig0MwbTyCbkjYbcbIn+nFyJcCxlZUskc7IjsjkS/bEkEats74sRTmTfM5IU\nvcEYAks0VG6EJ5MtYWQzJazSY2VJeIbKlrCEx229Z0C2hHVsZ044rXrHKE3oNFOHaSkUsWSa/liS\nIve0/ONpCsyAbIkxBfpirQAAEPFJREFU+sx0WhKz8iUG5EnkZEaE41aORE6uhKrLzZCwjhNKkGLJ\n7Hk0kSKeSh+UKOVDWBbWbtOwMiZEJmfCaRo4HSpHwpVz7MzJlrDzKOxsCdPKpTANgdPKnBiYQZHN\norBzJ3LzJ0wDK2fCclq13VVzcihyMyhMMcjBdZCTK1i5FEIMyJ2wnV8H/7fIZfBS81yn2dzsicH5\nFLmOtWnrTdn8CTuvgowzbm42RToNyX3yKrIv+1oyJ5cilU5nLNqHo1BRqD8EzgXiwDvAFVLKnjz3\nNQFBIAUkh7LAHczbe4Mc/s2/WyEx7oxJX22Jh9pidV5jnVcXuXE59JJMTWExDJGJLR1v8mVM2NkS\nsWSamCUogzMmMrkSqWy9Os/mSdh1dsZE3KqPJtIkU0kSKZnJoIincgN1rPdY5UzPlZho9jeVVqgo\n1LOAx6WUSSHE9wGklF/Nc18TsEpK2Xkgn79o6Qr55Z/9mT22SV9f1msp31/ASr/LEo6sSV9tiZtq\ny6yvxlo26nZo32KNZqKwA4mSaSVGuTkTdoLb4NyJ1KBf1Jlf39YvcvtXuJ0Ol/1Vn72e+4sfsr/s\npSRzjZzrA54okkzsXe6zd59eyaCeiX3dzpwQg3o4ptVzyk3iy82gsEOK7MAi+5rdu7J7ZioF0MAw\nsoFGdgLgpMujkFL+I+d0HXDRWH5+mc/Jp09ZuE99Oi3pDsfZ2xe1XsoKY28wSrt1/vruPjr7Y+T7\nQVPmc2bdXosHm/Npkz6NZiwxDIHLELjQPf5CMxkG8a8Efj/ENQn8QwghgVuklGuH+hAhxNXA1QCN\njY157zEMkVk1s2x26ZANSqUlXaFYxvnVFpSO/qhVZ5n0BWPEU/lN+ipsk74i5QCrPJXclklftq7C\n79I9FY1GM6kZN6EQQjwK1OW59A0p5X3WPd8AksBvhviYE6WUrUKIGuARIcSbUsqn8t1oichagFWr\nVo1qPM00hNVr8ABDC0quSV9nMEaHVdrmfJ39MTr642zvCNEVig05iVjscWREpcKvDPrs48oiZdaX\n8VrS7q8ajWaCGTehkFKeMdx1IcTHgfcBp8shJkqklK1W2S6EuBdYDeQVikJwoCZ94XgqY87X2a/8\nlLr6Y3RZnkqdwRhNndmMiqHm80o8DiqL3JT7nBnX1/IcIanwK3O+Sr+bcr+TIrdDL2PUaDQHTaFW\nPa0BvgKcIqUMD3GPHzCklEHr+Czgugls5pgihMBv7didW+nf7/2ptKQ3ksgIiW3Q190fpzuUrdsV\nCLN5lzLpS6TyK4vTFJT5XFRYpny5rq/lg+rKrLpS7QCr0WgsCjVH8RPAjRpOAlgnpfy0EGI2cKuU\n8hygFrjXuu4Afiul/FuB2jvh2PMcFX4Xi0dwf65JX3dI2YZ3hxJ0h2IZ075AWJn2bW3vz5j5DecA\nW+JxKrdXSziU62uOSV+OYV/uy+cydQ9Go5lGFGrV06Ih6ncD51jH24EVE9muqcxAk77991ggm1mh\nRMRyfQ0nlKCEE/TapWXg19wVyvgrDbfM3WGIjIiUeByUWKKiDPuyjq8lHseAOtuwTwuNRjO5mAyr\nnjQFIjez4kAcYNNpJTC2w6tyf82a89li0muZ9/VFErT2ROiLqON8K8VyMQ2RdX117+sAW5Tr/GqZ\n99nGfcVup2Xa59AbKTWaMUILheaAMYxs72XOQbw/mkhljPn6csTEdnkN5hj3BaPqemtPlGA0mDHz\n25/lACgTQeXkalLkdmYcX/3uHCfY3DrLsE+5wDrwW4Z9fpdyhdWrzTQzFS0UmgnHNrKr2X8+Ul6k\nlEQTaSUqluurLSD9sST9lqV4XzSh3F5jqYwTbFd/nJ1d4Yxx34E4wbpMA6/LxO8y8eW4wPpdjozj\nq+0I63OZeF0OvJYrbMYdNschNveaDl3STGa0UGimHEJkfZFqRvlZthNs2BKNjBNsPEl/LEUkroTG\nPg/HbVdYVR+Jp9jTFyViOcXa1w/Uq0gI8DiyAuJ2GspifIAbrKrPdYL1WM6vGRdYR9YB1nZ7ta9l\njwe6wur4Wc3+0EKhmdHkOsGOJfFkOuP+OtgR1nZ8tcUl1w1W3Zd1e40m1X2BUMJyh00Ts8poMjXk\nkugDwRDk2IorgXGaImMpbtuKK1dYYTnDWiJjCOUca2TrnYbAYTnDOi0HWIdpWI6wOQ6xlpOsaea6\nwxoD3WKta0aOR1GuP1GmzBxjOcbazrDTv5eW61M1wOsqTV4vrFSOZ1bWSXb4ecNpKRRvtgU596Zn\nVAZ0kZuqYpdVqvPqYnVcrDeiacYJ+xd7qXd02Rf7I5WWGeFQ7q8pywE2Tfz/t3d3MY6VdRzHf7+2\n03lddpddMcgS4QJfCBJikGsVA2iMBBMTjRHxJcgF4UZjxBVRFhJxNVyoMWpCNPHtzkgUhdXEeGFI\nUAMIAczGyMLqysLszE6n7+3fi+d5znna6XQ623ZOO/v7JOP0nJ5dn5MN85+2p9+24v3uvlCJjeuv\n8e1aVIiNK7GNlkugN9sW3e9uh2hfo5VWYSdFiOjFGXEX0nPRvpAfd2nxdH8c7Qv3BSHqB7hjthLS\n4kCUG48S453pcfcn4qS4WUiQ+8ih2YaI4bjtykGxNFfAgaUi/ne2imdPrm4a+SsWcm6ALBWTBtTB\nPUXfY0o7TQeXZrF3fkahP5k4+Rz96yJZryRlln7eQRgejXY7+Q222TY0/aAJv82G48Ox7a7PUej1\n23B3KbblP8MhlGLTH6i+FIvOiqxZ52/jyWdE2MYf4O683L6wo9fPZzPr+csnk//pHkDRdsdnYKBj\ngIWqbC4XDbyoNJvPMRl8nY+40vJseGSWFGajz/fI54DrH9z833RXDopD++fx409dl2y32oYzZd9f\nWqvjdKmK19bqvsXkkhr/Wa3imZOrWF6v97yiJrwB7sBiMQn7HVgM332TKbq9qPcCyHmKPmtdyLsL\nF2T67cpB0S0fVWN7Zgoj7bZhpdJIon6vlVyDySU0aji95r6/dGIdy6X6plfNFAu5JObX8bUQekxp\nOsOlNYq67l9EJtJ5MSi2IxelM97yxq2v36w2Wi7q5wfL6yWX0AhtptdLNSyXGzixXMZyqY61WnPT\nv2tptpC0lsL3MFDCvu5Gk4J/IjJuGhRDmpvJ45J987hk3/xAx9ebbayU3RAJ7SX3vY7lroTGS6+X\ncaZcx1p18+GSzzFpMO1bKGLf/Az2Lsxg33wYLGmDKW42XaDon4gMSINihxULOVx0gfvM7kE1W22s\n+ExGiPmdKdexWm5gpRK6TG7fqbNVvHBqDauVBkp9Hr0AwJ5Z12GKh0kc+wudpr1Rq8ndLujDlkTO\nIxoUU6CQz6WvsWxDo9XGSjntL61W6n7YbGwyrVQaOP5qKdlfa/a/xHG2kEvDfn6IhB5TGvlzHaY4\n+BeOWZot6BGNyJTQoNjFZvI5916SPdsbMIDvMXWF/c5W08ESekxnK+77SrmOl5fLyb5BrqVfLObT\nuF8I+0XdpRD6W5x18b/FpNuUHrM4W8DCTF6XLouMkQaF9JT0mLbxFFksDv+FFlN38K9Uc/vcd7d9\narWK9VrTNZxqzYHfTLSQBPzSkN/ibD6J+y2E7WK6vTC7eadpoaj6rEigQSFjMWz4D3BvXqo0Wij5\nIbJea0Uxv2Zyu1Rzjaakx+SHzOlSDeXlMsq1FtZ9g2mQ6mxQyLEj3jcfDZK5mc6oXxL7i267/b7T\nVMwnLaeOdlMhp0dDMvEyGxQkjwC4GUAbwKsAbvMfXNR93CcBfMVv3m9mP9m5VUqWyPCu48LQ8T/A\nDZ5as90R73MDJg36VeotrNddDDDsr9RbKDfSfaVaE6fXaqg20vuH6S4VCznMhcifjwDOFjZ+j2N/\nHbej0F8c/0v7TZ37k3ZTaDrlVa6V/rJ8RHHUzO4BAJJ3AfgqgDviA0heCOBeANfCvWP+byQfMbMz\nO71YmX4kkx/GFy6OvnnRaLWjsF8L1UY7Cf5VGi1U/UCpNkL0z91fa7g+UxIC9MG/WsP1lZbXXaep\n2nDdpXD8VhccbEcYGDNhkMQBwCQC6MJ8cQSw4I+J44AhCFjwkb+C3zeTT7MRhXxX/K8rBhi+cv5d\n3kl6IoT/usKAod8U9tPnLdx+9mw+dWYyJisg2NF5QpocacfJEZ8saflMiRmStIlZ7yBgnDxptjpz\nKP1kNijM7Gy0uYje6ZQbARwzs2UAIHkMwE0AftHv7z7+amlUyxQZWPihumduvCHAIDxCcsMjHSLx\ndoj+hVigC/oZ6s1Wcn/Dh/waPvbn9vtOUzu9HQKAlUorbTi12mi0o9st13FqRo2maRJHAAk3SMAk\n07QhBtg9W95+8QV4/r9nO/YlIUD/Iy7pR20IAlrfjxjOUqavUZB8AMCtAFYBvKfHIZcAeDnafsXv\n6/V33Q7gdr9ZIvniCJe6mYMAXtuB/5+dovOZfLvtnHbV+Tw/3efz5s3uoI2xUUvyD+hdVzpsZr+O\njrsbwJyZ3dv157/g99/vt+8BUDGzb41t0dtA8q9mdm3W6xgVnc/k223npPOZDmN9RGFm7xvw0J8B\neBTu9YjYSQDvjrYPAfjT0AsTEZGBZXahOMkros2bAbzQ47DHANxAcj/J/QBu8PtERGSHZPkaxTdI\nvhXu8tiX4K94InktgDvM7LNmtuwvo33S/5n7wgvbE+KHWS9gxHQ+k2+3nZPOZwqM9TUKERGZfmoU\niIhIXxoUIiLSlwbFEEh+jeRJkk/5rw9kvaZRIfl5kkbyYNZrGQbJIySf8f8+j5N8U9ZrGgbJoyRf\n8Of0K5L7sl7TsEh+hORzJNv+NcqpRPImki+SPE7yS1mvZ5Q0KIb3kJld478ezXoxo0DyUrgrzE5k\nvZYROGpmV5vZNQB+A5eKmWbHAFxlZlcD+CeAuzNezyg8C+DDAP6c9ULOFck8gO8BeD+AKwF8jOSV\n2a5qdDQopJeHAHwRvbMqU2XAVMzUMLPHzSx8dOETcO8tmmpm9ryZ7URJYZyuA3DczP5lZnUAv4S7\n7H9X0KAY3p3+aYCH/Xs9phrJmwGcNLOns17LqJB8gOTLAD6O6X9EEfs0gN9lvQgBsI3c0DTS51Fs\noV+GBMD3ARyB+y31CIBvw/3HO9G2OKcvwz3tNDW2SsWY2WEAh30q5k5sLABMlEHSNyQPA2jCVQ0m\n3qA5H5lMGhRbGDRDQvJHcM+BT7zNzonkOwBcDuBpn1w+BODvJK8zs1M7uMRtGUEqZqJsdT4kbwPw\nQQDX25S8EWob/0bT6iSAS6PtQ37frqCnnoZA8uJo8xa4F+Wmlpn9w8wuMrPLzOwyuIfP75zkIbGV\nAVMxU4PkTXCvH33IzMpZr0cSTwK4guTlJIsAPgrgkYzXNDJ6RDGcb5K8Bu6pp38D+Fy2y5EeeqZi\npth3AcwCOOYf9T1hZlN9TiRvAfAdAG8A8FuST5nZjRkva1vMrEnyTrgWXR7Aw2b2XMbLGhklPERE\npC899SQiIn1pUIiISF8aFCIi0pcGhYiI9KVBISIifWlQiOwgkr8nuUJyKt6cKQJoUIjstKMAPpH1\nIkS2Q4NCZAxIvsvHIudILvrPW7jKzP4IYC3r9Ylsh96ZLTIGZvYkyUcA3A9gHsBPzWyqEy9y/tKg\nEBmf++AaQFUAd2W8FpFzpqeeRMbnAIAlAHsAzGW8FpFzpkEhMj4/AHAPXN78wYzXInLO9NSTyBiQ\nvBVAw8x+7j9P+S8k3wvg6wDeBmCJ5CsAPmNmj2W5VpGtqB4rIiJ96aknERHpS4NCRET60qAQEZG+\nNChERKQvDQoREelLg0JERPrSoBARkb7+D/SPof1GaWmFAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NGwAplsagycR",
        "colab_type": "text"
      },
      "source": [
        "可以看到使用较小的学习率 $\\eta=0.4$ 和动量超参数 $\\gamma=0.5$ 时，动量法在竖直方向上的移动更加平滑，且在水平方向上更快逼近最优解。下面使用较大的学习率 $\\eta=0.6$ ，此时自变量也不再发散。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_3Lf9-eXD1aF",
        "colab_type": "code",
        "outputId": "799055c7-4710-4e7e-ac27-0327a1189877",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 298
        }
      },
      "source": [
        "eta=0.6\n",
        "show_trace_2d(f_2d,train_2d(momentum_2d))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "epoch 20,x1 0.007188,x2 0.002553\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEHCAYAAACncpHfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO2deZxcVZm/n1N7751OOmtn7xAIIQSI\ngAuKpBVQEUFWRxwVRZ1xmN+MM+MgDu46yqjjOoqO4zKKMioiigIJm+yELekQlmyQrbf03tW1n98f\n596qW53eu6puVfX7fOjPvffcW1WnQvf93nPO+35fpbVGEARBmH143O6AIAiC4A4iAIIgCLMUEQBB\nEIRZigiAIAjCLEUEQBAEYZYiAiAIgjBL8bn1wUqpEPAAELT68Wut9afGe828efP0ihUrCtA7QRCE\n8uHJJ5/s0lo3jmx3TQCAKHCO1npQKeUHHlRK/Ulr/ehYL1ixYgXbtm0rXA8FQRDKAKXUy6O1uyYA\n2mSgDVqHfutHstIEQRAKhKtrAEopr1LqGaADuFtr/Zib/REEQZhNuCoAWuuk1noj0AScrpRaP/Ia\npdQ1SqltSqltnZ2dhe+kIAhCmVIUUUBa617gXuC8Uc7dpLXepLXe1Nh4zBqGIAiCME1cEwClVKNS\nqt7arwDeBDzvVn8EQRBmG25GAS0CfqKU8mKE6Bat9R9c7I8gCMKsws0ooO3AKW59ftGx/RbY+lno\nOwh1TbD5Bthwmdu9EgShjHFzBCDYbL8Fbr8W4sPmuO+AOQYRAUEQ8kZRLALPerZ+NnPzt4kPm3ZB\nEIQ8IQJQDPQdnFq7IAhCDhABKAbqmqbWLgiCkANEAIqBzTeAvyK7zVdh2gVBEPKECEAxsOEyeNt/\nAirTdupVsgAsCEJeEQEoFpZsAjRc8E0I1UM87HaPBEEoc0QAioX2HWa76GRY/UbYvRW0mKMKgpA/\nRACKhfadoLzQeDw0t8DAEeh4zu1eCYJQxogAFAttrTBvDfhDsPoc07Z7i7t9EgShrBEBKBbaW2GB\n5YZduxjmnygCIAhCXhEBKAaGe4z9w0JHOYTmzfDyIxAdHPt1giAIM0AEoBho32m2C07KtDW3QCoO\n+//iTp8EQSh7RACKgbZWs3WOAJadCf4qEw0kCIKQB0QAioH2HVA5D6oXZNp8QVj5elkHEAQhb4gA\nFANtrebpX6ns9ubN0LMPju5xp1+CIJQ1IgBuk0xAx65MBJCT5hazlWkgQRDygAiA23TvgWQUFp50\n7LmGldCwWqaBBEHICyIAbtNmWUAsOHH0882bTSRQPFK4PgmCMCsQAXCb9lbw+GHe2tHPN7cYY7hX\nHilsvwRBKHtEANymrRUa14IvMPr5Fa8Db0CmgQRByDkiAG7jtIAYjUAVLH+NLAQLgpBzRADcZOio\ncf1cOI4AgJkG6twFfYcK0y9BEGYFIgBuYtcAGG8EAJlw0D0yChAEIXeIALhJ2gJilBBQJ43HQ+0S\nWQcQBCGniAC4SXsrVC+EqnnjX6eUCQfdc59JHBMEQcgBIgBu0t468fy/zerNEO2DQ9vy2ydBEGYN\nrgmAUmqpUupepdRzSqmdSqm/d6svrpCMQ+cLYyeAjWTV2aZkpEwDCYKQI9wcASSAj2mt1wFnAn+r\nlFrnYn8KS9eLkIxl1wAYj4p6aHqVCIAgCDnDNQHQWh/RWj9l7Q8Au4AlbvWn4IxWA2Aimlvg8NMw\n2JmfPgmCMKsoijUApdQK4BTgsVHOXaOU2qaU2tbZWUY3vvYd4A3C3DWTf03zZrPde29++iQIwqzC\ndQFQSlUDvwH+n9a6f+R5rfVNWutNWutNjY2Nhe9gvmhrhfnHg9c3+dcs2giVcyUrWBCEnOCqACil\n/Jib/8+11r91sy8Fp7118vP/Nh6PiQbasxVSqfz0SxCEWYObUUAK+G9gl9b6a271wxUG2mGoc2rz\n/zbNLea1bdtz3y9BEGYVbo4AXgtcBZyjlHrG+nmLi/0pHJO1gBiN1eeYrUQDCYIwQ6YwAZ1btNYP\nAmrCC8uR9p1mO9kcACfVjbDoZLMO8Pp/ym2/BEGYVbi+CDwraWs13j6VDdN7fXMLHHgMIn257Zcg\nCLMKEQA3mKgGwEQ0t4BOwt77c9cnQRBmHSIAhSYRNVnA01kAtml6FQRrZR1AEIQZIQJQaDqfh1Ri\nZiMArx9WvcGsA2idu74JgjCrEAEoNJOtATARzS3Qf9AYygmCIEwDEYBC094KvgpoWDWz91lt2UJI\nlTBBEKaJCEChadsBC9aBxzuz96lfaiqFyTqAIAjTRASgkGhtcgBmMv/vpLkF9j8EsXBu3k8QhFmF\nCEAhGTgCw925E4DV50AyCi8/lJv3EwRhViECUEimUwNgPJa/1qwnyDSQIAjTQASgkKQ9gKZhATEa\n/hCseJ0IgCAI00IEoJC0tUL9MgjV5e49m1vg6G7o3pe79xQEYVYgAlBIplMDYCKaW8xWwkEFQZgi\nIgCFIj5sntRzNf9vM3c11C+XKmGCIEwZEYBC0fEc6FTuIoBslDKjgH0PQCKW2/cWBKGsEQEoFLmO\nAHLS3AKxQWMRLQiCMElEAApF+04IVEP9ity/98qzwOOXaCBBEKaECEChaG+F+etMYfdcE6yBZWfK\nOoAgCFNCBKAQaG2mgPIx/WPTvNnkGfQfyd9nCIJQVogAFIK+AxDty/0CsJN0OOg9+fsMQRDKChGA\nQpCrGgDjsWA9VC+QdQBBECaNCEAhaG8FlFkDyBd2OOieeyCVzN/nCIJQNogAFIK2HdCwEoLV+f2c\n5s0Q6YVDT+X3cwRBKAtEAApBe2t+5/9tVr0RlEdsIQRBmBQiAPkmOmiM2vI5/29T2QBLTpN1AEEQ\nJoUIQL7p2AXowowAwKwDHHoSwt2F+TxBEEoWEYB8k+saABOxerPxHNp7b2E+TxCEksVVAVBK/Ugp\n1aGUanWzH3mlrRWCdaYOQCFYciqE6iUrWBCECXF7BPBj4DyX+5Bf2lvN079Shfk8j9fUCt69xWQg\nC4IgjIGrAqC1fgAo38nqVMqYwOXTAmI0mltgsN3KPxAEQRgdt0cAE6KUukYptU0pta2zs9Pt7kyN\n3v3GprlQC8A2zZvNVqKBBEEYh6IXAK31TVrrTVrrTY2NjW53Z2rkswbAeNQsNKUnZR1AEIRxKHoB\nKGnaW01iVj4tIMaieTO88ghEB6b1cq013UMxwrFEjjsmCEKx4HO7A2VNWyvMbQZ/ReE/u7kFHvpP\n2PcXOP4tY14WTSR5+WiYPR2D7O0aYk/nIPu6htjbOUTfcJxvv+sU3rZhcQE7LghCoXBVAJRSNwNn\nA/OUUgeBT2mt/9vNPuWU9laTmesGS88wFch2b4Hj30JvOMbujkH2dA5aW3OzP9AdJuUIFlpUF2Ll\nvCouOHkRK+dVc+LiOnf6LwhC3nFVALTWV7r5+Xkl0g+9L8Op7ynYR2qt6RiIsrtjkJfaBzgjtJG5\nT/+B8586j6PhePq6oM/DynlVrF9Sx4Ubl7C6sYrVjdWsnFdFVVAGhQVn+y2w9bPQdxDqmmDzDbDh\nMrd7JcwC5K89X7TvNNs8eQCFYwl2HRnguSP9PHe4nxfbB3ipfYD+SGbO/gOh4/kkD3LFmhj1S9fR\nPL+a1Y3VLJlTgddToLwEYXy23wK3XwvxYXPcd8Acg4iAkHdEAPKFHYOfgxDQvnCcnYf7aD3cx87D\n/bQe6mNv11A6z6uuws/ahTVccPJijltQw5r51TQvqKYxvg6++UP+efVBOPNtM+6HkAe2fjZz87eJ\nD5t2EQAhz4gA5Iu2HVAxB2qntoDaPRSj9VAfOw710XrI3PQPdGduEIvrQqxbXMcFJy9m3aJaTlxS\nx+K6EGrUTOOVZhF69xY488Mz/EJCzkklzRP/aPQdLGxfhFmJCEC+sGsAjGMB0ReOs+NQH9sP9bLj\nYB/bD/ZxqDdzs1/WUMmGJfVcefoyTlpSx4mL62ioCkytH80t8ORPzFOlG9FIwugMtMFvPjD2+bqm\nwvVFmLWIAOSDVBLan4NN78tqPtQ7zBP7unl8fzdP7OvmpY7B9LnlcyvZuKyeq169nA3Wzb6u0j/z\nvjS3wGPfg5cfzmQIC+6y51747QdNrYhT3ws7fpU9DaQ8ZiFYEPKMCEA+6N4LiWE6q9aw9fFXeHxf\nN4/t604/3VcHfZy2fA4XblzMxqVzWL+klvrKKT7ZT5blrwVv0GQFiwC4SyoJ9/07PHAjNK6Fv74d\n5p8AK16biQIK1ZmynkpyNIX8IwKQI7TW7Okc5JG93cSevZ2rgffeMcxOvYO5VQFOX9nA1a9byekr\nGzhhUW3honACleYGI2Ui3aX/iJnyeflB2PhueMtXIFBlzm24LLPgm0rCD86BO6+H486FYI17fRbK\nHhGAGXC4d5h7X+jg4d1HeWzfUboGYwB8qnInSTxcdcG5bGpeyOrGqjEWaQtEcwvc+QnoPQD1S93r\nx2xl91b47TUQD8M7/gs2vmvsaz1eeOvX4IebzWjh3C8Urp/CrEMEYAqkUprth/rYuqudLbs62HWk\nHzDZs69f08iZq+ZyxqoGlt35E1TPWq54TbPLPbZYbU397NkKp73X1a7MKpIJuO9L8JevQuPxcOmP\nYf7xE7+u6TSTQPioJRaFqiYnzDpEACYgEk/y4EtdbNnVztbnO+gciOJRsGl5A9edfzybT5jP6sbq\n7Cf8tp2w/NXudXokjWuhtsmEg4oAFIb+w9aUz0Nwyrvh/BvNdNxkafk07Lod/vhP8L47CldQSJhV\niACMQtdglHt2dXD3rnb+8lInkXiKmqCP169tpOWE+Zx93HzmjBWOGe6G/oOFrwEwHkqZBeCdt0Iy\nDt4cRBcJY7N7izXlE4GLboKTL5/6e1Q2GBG4/Vp49pewsXxdUwT3EAGwiMST/Lm1jVu2HeCRvUfR\n2iRdXb5pKS3rFnDGyrkEfJOIzEhbQBSRAIBZB3jqJ3DwCVj+Grd7U54kE3DvF+DBrxkL8Et/Ao3H\nTf/9TrkKnv4Z3P1vsPZ8qKjPXV8FgVkuAFprth/s45ZtB/j9s4cZiCRY2lDB352zhjevW8CJi2un\nvnibtoDIjwfQtFn1BlBe83QqApB7+g7Bb642NRhOfQ+c9+WpTfmMhscDb/0q3HQ23PN5eOt/5KSr\ngmAzKwWgZyjGrU8f4ldPHOCF9gFCfg/nr1/EpZuaOHPlXDwzCdFsa4WqRqhZkLsO54JQnbGI3r1F\nkoxyzUt3mymfZAwu/iFsuDR3773oZHjVB+CJH5q1hMUbc/fewqxn1giA1prH9nXz88de4c7WNmLJ\nFCc31fH5d6zn7RsXUxvK0bx4+47imv930rwZ7vkcDHZA9Xy3e1P6JBNw7+fhwa+b/+eX/hjmrcn9\n57zxerN+88ePwdV3m5GBIOSAWSEAj+49ytfufpHH93VTV+HnXWcs4/JXLeWERbW5/aBkAjqehzOu\nye375ormFiMAe+6d3sKkkKHvIPz6ajjwKJz2PjjvS/nzWqqohzd9Dn73YbMmcNpf5+dzhFlHWQvA\nky/38LW7X+Ch3UdprAny6QvWccXpywj5vfn5wKO7IRktvvl/m4UbzPTU7i0iADPhxTvh1g+ZiKp3\n/jecdEn+P/PkK+Cpn8KWT8MJF5goIUGYIWUpANsP9vK1u1/kvhc6mVsV4JNvPYF3n7k8fzd+m/QC\ncJEm7ng8sPocIwCplEwlTJVk3Hj2PPxNI/KX/hjmFSjZTymzCPy9s4wIvP2bhflcoawpKwHoDcf4\n5O9a+cP2I9RX+vn4ecfznlcvL1yZw7Yd4PHDvBmE/uWb5hbY/is48gwsOdXt3pQOvQfg1++Hg4/D\npvfDuV8Cf6iwfVhwIpz5EXjkOybSqGlTYT9/JkjZy6KkbATg8X3d/L9fPk3nYJRrN6/hg2etpCZX\nC7uTpb3VpPz78uTsmQtWnwMo408jAjA5XvizmX9PJuCS/4H1F7vXl7P/FVp/A3/8R/jgvcY7qNiR\nspdFS8nPASRTmm9seYkrbnqEgM/Dbz7yGv7xTccV/uYPJgS02BLARlI1z4QS7t7idk+Kn2Qc7vok\n3Hw51C2FD93v7s0fjDvouV+AI8/Cth+525fJMl7ZS8FVSloA2voivOsHj/L1LS9y4cYl/OHas9jQ\n5FK25FAXDLYVbwiok+YWM5Ux3ON2T4qX3lfgf86Hh79l4vCvvhvmrna7V4YTL4aVb7BCejvd7s3E\njFXesu8APPo92PcXY6HiBttvga+vh0/Xm+32W9zph0uU7BRQz1CMS7//MEcHY3z10pN552kul9Br\n22G2xT4CACMAD9wIe++HE9/hdm+Kj+fvgN99xHjzX/pjOPEit3uUjVLwlv+A/3oN3H0DXPRfbvdo\nfGoWwcDhY9uVB/78ccd1i806x8L15kFqwYmmprXTuyqXawkyNVWaApBIpvjozU/R3hflVx86k1OW\nzXG7S8VrATEaSzZBsM5MA4kAZEjEYOtn4JFvmwzcS38MDavc7tXoNB4Hr/moSUI79T3F5T7rJJkA\nf9Wx7f4KuOCbZiTT3mo8tOyfvfdBKm6u8waMm+2C9WZKbtftJtQaZn7DHm9qSgSgePn3Pz3PQ7uP\n8pVLNhTHzR/ML27NIqia63ZPJsbrg9Vnw557QGuxGgboeRl+/T449CSc/iF48+fAF3S7V+Pz+n+G\nHb82GcIfesD8fy027vsSdL8Em66Gl+4a/cm9ZkF2udJEzOTUtO80mfW2KAwcOfb9Z3LDHnNqapT2\nMo1iGvc3RilVCzRqrfeMaN+gtd6e156Nwe3PHuaHD+7jva9ZwWWbiqi6VVtracz/2zS3wHO3Qefz\npi7tbOb5P5opH63hsp/Cugvd7tHkCFSZDORfvRse/z68+m/d7lE2e+4xxXBOeTe87WuTf50vAAvW\nmR8cvkqfrgf0sdePdSOfiLomM4oYrd1JGU8VjbkIrJS6DHge+I1SaqdS6lWO0z/OxYcrpc5TSr2g\nlNqtlPrXybzmfx7ax9oFNVz/1iK6aSVi5kZarAlgo2FXCZvN0UCJGPz5Ovjlu8xUz4ceKJ2bv83x\nb4PmN8G9XzJ1h4uFgTZjkNd4vCmGkwtG3pgnap+Icz4JjDL6PeWq7OMyjmIaLwroE8BpWuuNwPuA\nnyml7NWwGc8ZKKW8wHeA84F1wJVKqXXjvSapNc8e7ONN6xbg9xZRAFPXi2bOcmEJzP/b1C2BxhNm\nrwD07IcfnQuPfhfO+DC8/05oWOl2r6aOUqbAfDJmQlaLgVTSVEOLDZl1lJnaYttsvmF0v6UNV0zv\n/RIRQENFA6DMInTFXHjoG/DVtZnIoNFGCTB2ewkx3hSQV2t9BEBr/bhS6o3AH5RSSxl1HDZlTgd2\na633AiilfglcCDw31guGIgmSKc3r1szLwcfnkPQCcAlNAYGZd338JvOHGhhloa6ccM7hVjZANGym\nGi7/X+OtU8o0rILX/T+4/8tmQXjVG9ztzwM3wv6/wIXfnVwN5MliT7fY/x9rFxlLk8e/D+veDos2\nTP69wt2w5TOw7DXZJTcf/pYR0viQOZ7oJr/9lpKeBhrvMXpAKZUOfLbE4GzMTToXcx1LAOe/7kGr\nbUyGYkl8HsWpxbLwa9O2A7xBE7JWSjS3mCfH/Q+63ZP8Ys/h9h0ANISPQioCZ3+89G/+Nq/7B6hf\nDnf8k5nacot9D8B9/w4nXwmn/FXu33/DZfAPrfDpXvjHXfDBrSY57n/fCd17J/8+934BIr3wlhuz\ngyAe+/7U+lPi00DjCcBHAI9zWkZrPQCcB3wg3x2zUUpdo5TappTaFouESaQ0h3qHJ35hIWlvNQup\nxRiFMR7LXg3+yvKfBhptDldreLTI4+engr/C3My6XoRHv+NOHwY7zNTPvDUmT6EQ1DXBVbdCKgE/\nu8isPUyEnUX9qg8em7cz1QXl6S5AFwljCoDW+lmt9UvALUqpjytDBfA14G9y8NmHAGcYT5PVNrIf\nN2mtN2mtNy2ca7J8H3ypiLIftS4NC4jR8IdgxVnlLwBTCfcrZY47F9a+Fe7/ijGvKySplFn0jfSZ\nef9gdeE+u3Et/NWvTVb0/14Cw71jX6s13PHPZt7/jZ849vxUF5SnuwBdJExmJfUMzI36YeAJ4DDw\n2hx89hPAGqXUSqVUALgC+P14Lwj6PCypr+CBl7py8PE5YrADwl2lkQA2Gs0tZuh8dM/E15YqoTEK\n/5T4H++onP/v5iZ353WF/dwHvwp774Xzv+xONFzTaXD5z0w03s1XHjvis9n+KzjwGLR82hTaGclY\nC81jUeLlVScjAHFgGKgAQsA+rXVqph+stU4AHwXuBHYBt2itd070uvPWL2TLrnbue6Fjpl3IDe0l\nZAExGnYCzp573O1Hvnj+j+apVI1wzfRXlPwf76jUL4PX/5PJmH2pQCO7/Q/BvV+E9ZfAqS5WK2ve\nDBd/H155xFh3JxPZ5yP9cNe/wZLTYOMY6xMbLjMZynWTyDHadHVJLwDD5ATgCYwAvAo4CxOu+X+5\n+HCt9R1a6+O01qu11l+YzGs+9ubjOH5hLdfe/DT7uoZy0Y2Z0VbkRWAmYu5qmLPC2EOXG2074Dcf\nNH/wb7f/qJXZXvDNkv/jHZPX/J0JSLjjnyAeye9nDXXBb66GOSvhgv90P6t8/TvNWsgLd8Dtf29G\nQzb3fxmGOs358Yoh2QvNF/9g9NFARYM5N5XktiJlMquWV2utt1n7R4ALlVJXjfeCfFIZ8HHTVafx\n9m8/yDU/3catf/taqgtV8GU02luhtgkqiiwyaSo0t8AzN0MiWvz2B5NlsNNMBYTq4IpfQM1Ck5E6\nG/AFzU3uZxeZ6mVv+Jf8fE4qBbd+2IRUfuAWE41TDJz+QSNM9/+7ueF3PGet92hYfpZ5IJgMI8NO\ny8gCwmbCEYDj5u9s+1l+ujM5ljZU8p13ncreriE++JNtdA5E3etMqS4AO2luMXHPrzzqdk9yQyJq\n7BGGuuBK6+Y/21h9Dqx7h7Fi6Nmfn894+Juw+24474tTi8EvBGf/K6w8G166MxP+C3Bo29Qsn51h\np//QWlY3fyjhegCvaZ7HjZds4KlXejj/Gw/wwIsuRAbFIybsrtQSwEay4ixTyrIcooG0hj/8Axx4\n1NgkLz7F7R65x7lfNGsff/r4xNdOlVceM0/G695h5sKLDaWge/ex7YnysHDIFSUrAAAXn9rE7z/6\nOhqqArznR4/zpTt2EUvMeH168nQ+DzpZ+iOAYLWxEy6HdYBHvg3P/BzOvq74fPwLTd0S8yT84p9N\njYNcEe42i6z1S83aitvz/mPRd0xUudVeZuG/M6CkBQBg7cIabvvb1/GuM5bx/Qf2cun3HmZP52Bh\nPryUagBMRHMLdOyE/lEKd5QKL95pojzWvQNen6d571LjzI8Yz6c/fRxi4Zm/n9bwu7+BwXZTHzlU\nN/P3zBe5No8rQ0peAAAqAl6+eNFJfPevTmVf1xBv/voD/Muvn+VAdw5+4cejrdVk0paiidhImlvM\ntlRHAR274NdXm7nod/zX+FEeswmvH976H9D3CjyYg6iVR74DL/4J3vx5WHLqzN8vn4wW01+u4b/T\npKz+St5y0iK2fOwNXHXmcn739GHO+ep9fPJ3O2jry1MoXHsrzF8HHu/E1xY789eZgjaluA4wdBR+\ncblxnbzi5ty5T5YLK14HGy43LpczSfg7uA22fMpYUJ/xodz1L19kxfTPgvDfaaC0zoWxZ2HYtGmT\n3rbtmKCkUTnSN8y379nNLdsOoJTir85YxkfOXs38mlBuOqM1fGWl8Y+/4Bu5eU+3ue1vTQLRP+8t\nHV+jRMyEOx58wrg6Nm1yu0fFyUA7fHuT+fd592+nPm8/3APfe70xgv/QA6Ud9jwLUUo9qbU+5o+j\nrEYAThbVVfCFi07ino+dzTs2Luanj7zMWV++l0/cuoMdB/uYsfD1HzZ/FKUeAeRk9WaTNXv4Kbd7\nMjm0NslOLz8IF35Hbv7jUbMA3ni9yfh+7rapvVZruO2jprD7Jf8jN/8yomwFwGZpQyVfueRktv7j\nG3j7yYv5zZMHueDbD/LWbz7ITx/ZT184Pr03LtUaAOOx6mxQntKZBnrs+/DUT+Csj8GGSye+frbz\nqg+YokV/vg6iUwiUePwmeP4P0PIZEdkyo+wFwGbFvCpuvPRkHr++hc9deCIeD9xw205O/+IW/v6X\nT/PQ7i5SqSmMCtosD6BStYAYjcoGWLKpNARg91ZjeHb82+CNRVIJq9jx+uCtXzNP8vd/eXKvOfy0\nKZBy3PnFV3NYmDElMtGbO+oq/Fz16hVc9eoVtB7q4/+2HeB3zxzmtmcOs6S+gks3NXHxKU0smzvB\nQmJ7qynAMZbTZKnS3AL3fcksrFbNdbs3o9P5Ivzf+8zC9UXfl4ifqbD0dGOJ8eh3YeO7TB2LsYj0\nwf+9F6rmwzu+W7zx/sK0mdV/OeuX1PGZC9fz2Cc2880rT2FVYxXf2PoSr7/xXs77zwf46l0vsP1g\n7+gjg7bW0qoBPFmaWwBtrH2LkXA33Hy5Ked45c2F9Z0vF1o+A4Fq44s/1lqY1vD7a01dgUt+ZEaH\nQtkx60YAoxHye3n7yYt5+8mLOdQ7zJ92HOHu59r5zr27+dY9u1lQG6TlhAW0nLCAV6+eS0hHoXuP\ncR4sNxZvNG6Hu7fASZe43ZtsknHzRNp3EP76D8b6WJg6VfOg5VPGMmPHr0dfP9n23/Dc74xv/rIz\nCt1DoUCIAIxgSX0FHzhrFR84axU9QzHufaGDu59r59anD/Hzx16hwu/l3Uu7uF6n6K1byyglJUob\nj9cYie3eatwei2l65c/Xwb77TbFxuSnNjFP/Gp76Kdx1PRz35uyM3iPb4c+fgOY3wWv+3r0+CnlH\nBGAc5lQFuPjUJi4+tYloIsmje7vZuqsd7477Abjg/3ppePhBXn9cI2esnMtpy+dQESiDpLDmFmj9\ntSl2s+hkt3tjeOKH8MQPjNd9PoqNzzY8XrMg/INz4N4vmUpiANEBM8qqbICLvldcDwBCzhEBmCRB\nn5c3HNfIG45rRPviJJ+p5vKzXss9L3Tx3fv28K17duP3Kk5uqueMVQ2cbgmCq7UKpsvqc8x295bi\nEIC998Md/wJrzjXz10JuWM0uZhQAACAASURBVHIqbHofPPZfsPO3prypvwLiYXjvHWaqSChrSvDu\n5D6qfSfehev56Oa1fHTzWgajCbbt7+bRvd08uvco37t/L9+5dw9ej2L94lrOWDWX01c0sGnFHOor\nA253f2JqFpgF7t33mBh7Nzm6B255D8xbA+/8YXnYbhQTdiDDYLvZxsPGGrx/DCdNoawQAZgqWkP7\nzqwF0uqgj7PXzufstfMBGIomeOqVHh7b283j+7r58UP7uemBvQAcv7CGV61o4FUrGzh9RQML63Jk\nTZFrmlvg4W+ZOqpuhboO98LNV5jktCt/WX4ht8XAX0YxiEvFjWe+eOaUPSIAU6X3FYj2jVsDoCro\n46w1jZy1phGASDzJswd6eXxfN4/v7+a3Tx3kZ4++DMD8miAbmuo4aUm92TbVMa+6CMoyNrfAg1+H\nfQ/ACW8r/OcnE8ZzvnsvvOe28nBcLUbG8sYXz/xZgQjAVJlGDYCQ38sZq+ZyxiqTWJVIpth1ZIAn\n9nez41AfOw71sfX5jnRI9qK6EOuX1LF+cR0nNdWyfkld7kzsJkvT6RCoMesAbgjA3TfAnq3GaG/F\n6wr/+bOFuiarZOIo7ULZIwIwVdpaAQUL1k37LXxeDydZT/s2g9EEOy0xaLW2W3a1p0Vhfk2QExfX\ncuLiOk5cbEShaU4FKl/Zmb4ArHqDCQfVurBZoE/9FB79DpzxETjtvYX73NnI5hvg9mshPpxpE8/8\nWYMIwFRp3wENqyBQldO3rQ76skYJYNYSnjvSz46Dfew83M/Ow3088FIXSSszuSbkY90iIwprF1az\nZkENa+ZXUxPy56ZTzZuNCVjXS9B4XG7ecyL2PwR/+EcTifTmzxfmM2cz9jz/1s+aaZ+6JnPzl/n/\nWYEIwFRpazVVpwpAVdBnFoxXZNLwI/EkL7QN0HrYiMJzh/v5xeMvE4lnaiEvrA2xZkE1qxurWbOg\nmubGaprnVzN3qmsLqzeb7e4thRGAnv1wy1UwZ4WxHS6VmgSlzobL5IY/S5G/sKkQHYSefbDRvUSk\nkN/LyUvrOXlpJgc5mdIc7AnzYvsgL3UMsLtjkN0dg9yy7QDhWDJ93ZxKP6stMVjdWM3q+VWsbqym\naU4lXs8oUzxzlsO844wAvPpv8vvFIv3wiysglYR3/Qoqyi7HWhCKDhGAqdDxnNmOEwHkBl6PYvnc\nKpbPreJN6xak21MpzZH+CC+1G1HY0znEno5B7nqune6hzMJfwOthxbxKVjdWs6qxipXzzHb1vGrq\nmltg24/MHPHI+qq5IpWE334Qul6Ed/8G5q7Oz+cIgpCFCMBUSNcAKC4BGAuPR7GkvoIl9RXpHAWb\n7qEYezsH2ds5xJ7OQfZ0DvJC2wB3PdeeXmMAeGtFPd/REb77k58w2HQ2yxoqWdZQydKGShbVhfB5\nc2AVsPUz8OKf4S3/AavfOPP3EwRhUrgiAEqpS4FPAycAp2utJ1fo123aW41pVhmEyDVUBWioamDT\nimyb33gyxYHuMHs7h9jXNcQrHQuItf4H89oe5Gt7l5FwiIPXo1hYG2JpQwVNcypZOqeSpjkVNM2p\nYMmcChbWTkIgnrnZFCvfdDWc/sF8fFVBEMbArRFAK3Ax8H2XPn96tLWap/8yLozh93pY1VjNqkbb\nZ38VDJ/FZX0vcPF153GkL8KB7jAHesIc7BnmQLfZ/uWlTtr7o1nv5fUoFtWF0qOQJXNGbAe3E7z9\nWlj5ejh/khWqBEHIGa4IgNZ6F5C/GPZ8kEoZC4hT3u12TwpPcwvceR2+/gMsbVjO0obRq6VF4kmO\n9EU42BPmUM8wB3uGzX7vMI/uPUpbfwR7ALGYLm4LfpKIauBfev+G2l88y6K6ChbXh1hYZ0YPC2qD\nLKgNEfKL/48g5IOiXwNQSl0DXAOwbJmLBUB69kF8qOgWgAuCJQDs2Qqb3j/mZSG/l5Xzqlg5b/Qc\niXgyRVtfhCOdXRz3x0uoGErys9X/QTA6l31dQzz4UhdDjqglm9qQj/mWIMyvCTG/JkhjTZD5tSEa\nq4PMrzXHNUFfaT1UCILL5E0AlFJbgIWjnLpea33bZN9Ha30TcBPApk2bplC1PcekLSBmoQDMWwN1\ny0xW8DgCMBF+r4el9SGW3nUdDLwE7/o//nFNS9Y1/ZE4R3ojdAxEaO+P0t4foaM/QseA2X9ifzcd\n/VFiydQx7x/ye5hXbcSgsTrIvJqgOa4OMM9xPK86QLWIhSDkTwC01i0TX1VCtO80rpTjFdEuV5Qy\nWcE7fg2JmLGJmC73fdFkF5/7JVhz7K9IbchP7UI/axfWjPkWWmv6hxN0DEToHIjSMRClYyBC12CM\nzoEoXYNRXukO8+TLPXSHY6OWvQ36jFjMrQ4wtypAQ5URBrM4HrDag+njyoBXBEMoO4p+CqhoaGuF\nuWvyFwtf7DS3wJP/Awcfn745245fwwM3wilXwZkfmXZXlFLUVfqpq/SzZsHYQgHGeK87HKNrIEbX\nYDT9c3QwRtdgjKNDUboGYzzfNsDRoRixxLEjC4CAz8PcqgBzKgPMqfJTXxGgrtJPfYWfugo/9ZVm\nW2sd2z8y0hCKGbfCQC8CvgU0An9USj2jtT7Xjb5MmvYd0PQqt3vhHitfDx6fyQqejgAcfBJ+9zew\n7DWmFGGBboo+r8daN5jYTVVrzVAsydHBKEeHYnQPxugOx+geitEzFOOote0Jx2jr66dvOE5vOJ4V\nGjsSj4LaCj+1IVsgfGaUE/JTE/JRW2G2NSF760ufqwkZAQn4pCyjkB/cigK6FbjVjc+eFpE+Uwfg\ntPe53RP3CNXC0jOMALR8emqv7T8Mv3yXqTR2+c9mNoWUR5RSVAd9VAd9LJ87ObM/rTXhWJKecIy+\n4Th9w3H6ra3ZT5htJNPe0R+lPxJnIJLIsuoYi4DPQ03QR3XI9K3K6mNm35tuqwz4qAp6qQr4qLS2\nVUGvaQ/4qAh4RVCENDIFNBnad5rtbFwAdtK82bhGDrSbm/lkiIXh5ishNghX3V12dWaVUlRZN+Km\nOVN/fTyZYjCSYCCSSIvCgGM7FEvSH4kzGEkwGDXXDUYTtPdH2BtNMBhNMhRNMByfWEhsfB5FRcBL\nZcAIQ8hv9iv8XirsrbUf8nsJ+T1U+DP7Ib+XoM9D0O8l5PMS9HsIeD2E/B4C3sxxwOfB7/Xg9yqZ\nBitSSkoAOgei/Ln1CCvnVbN8bmXh4sPbrAig2RgC6qS5xQjAnntg45UTX6813PY3cORZuPLmGdVQ\nKFf8Xg9zqgLMqZrZqCiZ0gzFEoSjSQajCYaiZnQRjiUYiiUZjiUYippj055kOJYkHDfnhuPmXNdg\nlEjcnI/Ek0QSqTHXRaZCwOcUBYXfa4591r7P68HvUZljj8Ln2Po9Cq913utR+DwePMoce5TC6wGv\nUng8KrP1KDwKPEpZP8YeRSmFwrQrZabpFArrv/R5m5HaZQcVaMwIUFsHGo3Wpj2ltcl5sbYprUmm\nzPmk1uZ8SpNMWccpndmmNImsbSpznNTEkymSKU08pUkkUySSmlgyRSKVSp+Pp7cpNi4d21ixpASg\nrT/Ch//3KcD8T1lcV2GZl1WxYm4VKxurWDm3iqY5FbnxqLFp3wEVDVCzKHfvWYosOAmq5ptpoMkI\nwP1fgZ23QstnYO35+e/fLMbrUem1hVyTTGmiCSMY0UTKCEM8RSRhRCJqiURmmySWMDefWCJFLKnN\nNmFuUqbd3LRiiaS5aTluZoOJRNaNLpHS6Ztb8pgbZIqUdRNNjrMWU+x4FJZgKXxpsfNkHfu99tZj\nCWFGNGv8vrRw+m2x9Xrw+xSr5lXz32N8bkkJwImLa/nJR1/H3q5B9nUNpX9ufeoQA9FE+jqfR7G0\noZIVcytZYYmD2VaypH4a4tDWap7+Z/sw1uMx00Av3mkcPD3jjMB2/s6EfJ58Jbz27wvXRyHneD2K\nyoBZXyh27Cfp0Z62U9o8sdtb+0ld68wWyAobtp7vjymKZ48R7DalMiMHpcxoBMfoQ8GooxL7ON9T\nZFeP0V78/0cdeJQ6ppQimP+ZR4di7O8aYm/XEPu7hth/dIj9XWEe29edtdDm8yia5lRY9snGvGzp\nHONuuXROJbUVI8L2Ukno2DWjBKiyorkFnr0ZDj8DTaeNfs3hZ+DWD5u6wm/7TxFOoWB4PAoPCnEP\nmRwlJQBjoZSyMjyDx7hbaq3pHIiyr2uIl7vDvHx0iP1HzfapV3oYiCSyrq8J+mhqyLhanhjo4JLE\nMAdDq6gJx48ViNnGqjcCykwDjSYAA20m4qdyLlzxc/AXuJi9IAiTpiwEYDyUUsyvDTG/NpRVb9em\nLxy3nC0z7pYHeoZ55WiYh3Z38cbEQ1wSgGvujPLcn++iJuhjiW15nHa2rGRxfYglcyqYVxXEM1p1\nrXKhai4sOdUIwNkfzz4Xj8Av/wqGe+D9d0L1/NHfQxCEoqDsBWAiTEZpHeuX1B1zTmtN5M5H0I/5\n+PvL38aBgaTlcGlcLh/b25219gAm2mFxXYjF9RUstm2Q6ytYVB9Ku12WwlzquDS3mIzecDdUWiMu\nreH3fweHtsFlPytY3WRBEKZPid+J8otSioqju6BxLeduXD7qNf2ROId6hs1Pr/k5bP08+FIX7QOR\nY7xoakM+FtaFWFAbYmFtKL1vHy+oCzK3Kjh6nd5iYPVmuP/LsPc+WH+xaXvw67DjFjjnk7Du7a52\nTxCEySECMBHtrbD8tWOerg35qV3k54RFtaOejyVStPdHjCj0DXOkL0J7X4QjfRHa+iO82D5A50CU\nkRFsXo+isTqY9sSfXxtkQY3Zzq8NpfcbKgOFn3JacpqpjLZnqxGA5/9o8gPWvxPO+qfC9kUQhGkj\nAjAe4W7oPzSjBLCAz2MijMYoogLGsKxrMEZ7vxGFjv6MFXJbf4T9R4d4fH83veH4Ma/1ecwC+Pza\noOWT7/DLd/jmz6sOEPTlKDTC64OGZnjmF/D0z01b/TK48DsS8SMIJYQIwHgUqAaAz+thYZ2ZCjp5\nnOsi8WTG/tjyyO8YiNDRH6V9IMrBnmGefqV3TAvk2pCPRtsj37GdWxVgbpY18gR++dtvgfbtoB0Z\nooPtsOt22HDZjP4tBEEoHCIA45G2gDjJ3X5YhPzeCUcTYPxluodidPRH6Rw0AtE1GLW88o1n/s7D\n/XQNRI9ZxLYJeD1Z3vgNlhXy3KoA73viBqqTI0YjiYiZBhIBEISSQQRgPNpbjfVBiYUz+r2e9KIy\nHBvd5CQST6atj7uGonRbHvlpO2TLBvmV7jDdgzEGogn+NngERhkcpPoO8sYb7zWe+ZV+43Fj7ddX\njti3PPUrApKxIwhuIQIwHu2tZW8AF/J706GqkyGWSKG/0QQDB4851++fz8lN9fSEY3QORnmxfZCe\ncGxcy+Ogz0N9ZXaBFbu4Sn1lIF1gpX5EoZXaCn/xRkkJQokgAjAWyQR0PA9nfMjtnhQVAZ8H3vQp\nuP1aiA9nTvgrqL/g83xzwynHvCYST9I3HKcnHKNnKE5vOEavVUylNxyjN2zO9Q7HeaU7zPaDxjd/\nIovjmmCmoMqxBVacRVb81ARNW7Wj0Ep10CciIsxqRADG4uhLkIwWzfx/UWHP82/9LPQdhLom2HzD\nmPP/IctL3kxJTZ5IPJlVXMWuwOU8tn30+4fjHOmL8GLHgOWln5iUO2RlwGvEIJRdZKXG2joLrlSN\nKMJSGbCLsJjzQZ9ndtuECCWHCMBYtBUmAqhk2XBZ3hd8beGYP0XhAJPFPRxPpsVgIBJPF1Qx+0mz\nHVFoZSiaoHsonN4fiiaJJSfnh+/1KCr93nQlrsqgl0q/qcJVFfRS4fdZRVi86YIsFQGfeU3AS8gq\nxmIXZwk5CrSE/F4ZrQg5RwRgLNp3gDcA89a43RNhGiiVsTBeMHqO3qSJJVIMRS2BiGWEYShqiq3Y\n54ZjyXRRFvs6u1zkod5k5nwsOa0iK3bVLVsYQr5MhS67Spezale6cpfPQ9Cq3BX0eaziLN7MfrrN\nnPd7M2124Ra/V+W2xoZQFJSUAPQPx3n6lR4W1oVorA7m9xeyrRUa14I39wU2hNLC3AxnXrXLSSKZ\nYjieND92ha54kohjfziWJGIVYonEzfUR6yezb9p7w7F0sZaR21zVSfEorBKPmapefoc4+Dwq69g/\nomiJfY3Xo/B7PHi9juInHnOt1wNeT+a6dLUvjwevVdHL6aOf2bere2XOgcN73zN61S9lVQOz7PsN\nKuP3D+NVBMuUBtMj2tM1BtLnNakUjhoEznoFZsSatGoWmNoFmQphKU26EI5d6yBlFcVJaZ1VLSyr\nepi1XTVv7PrWJSUAL3eHuei7DwPml3FedXAMT50gCy0H0NrQNO2b21uN540g5AGf10ON10NNHip4\njSSeNNW6oo4Sj87KXdFEyqrOlcqq5BVPZqp5pdtSKeIJU60rkTKVvez9eNJU9bLLEUbjKeKpZLrS\nVzyVIpXSxJM6q9LXyCpfpVzZq9B4FPg8HryOYjN2xTC7XY+WFWpRUgLQPL+ab/71Jtr6jZ9OW7/x\n1HnlaJjH93XTN3ysVUKF38sC2z+n1tgkpP11bG+dmmB25utgp8lsLfMQUGF2YD+pVwdL48/dfho2\nT8CZco/JVHZ1r8wTcuYpWVtPzs4qX/aTtl0FTFuP5drx5G7fJJ1P8tZVpE+MVxEMa1RhHdsjEOUY\ncdijEeeowx69ZNUu9jjbMzd2r2NU5LPPT3Jd6BtjVHAtjd8Iiwq/l80nLBjz/HAsmfHTsewSzHGU\n9r4IOw720t4fHTW8sMLvTYvBWZ4dXAv89vAcoo+/YhWbCaStEwpWjF4QZiHKKvZeUjenEqWs/o0r\nAl5T+3ecOS+tNYPRRNpszfbS6XB47ASO7gLgs0946H1ixzHvUR30pQVhnu2hUx2k0drOqw7SUBVg\nXnWAugq/hAYKglCUlJUATAallJUg5Kd5fvXoF/32x7BvMY998lKOWt45XYP2TyyzHYiyt2uQx/fH\n6BnDgM3nUZafjm26FrDEIZjx2klvg1JyUhCEgjHrBGBSWBYQQZ83XdlrIhLJFN3hGF0Dtn+OEYmj\nlnB0D8XoGrQ8dYZiDI5hwubzKOZUBWioNKLQUO3Yrwqkz9WnvXb8VPi9IhqCIEwZEYCRJGLQ+QKs\nefOUXubzesyics3kkpYi8STdQxmzte6hKEct8zXnz67D/Rwdio26wG0T8HnSHjr1lYH0/pxK218n\nYHnrZLx06iv941s+C4JQ9rgiAEqpG4ELgBiwB3if1rrXjb5ksf0WuOuTkIrDkz+B+SfkLds15J/8\n6ALMCMP20zk6GKMnnPHU6QnH6LP8dHrCxk/n2YNmf7yEI69HZZmr1af3fZn2UOa887g6JD46glDq\nuDUCuBu4TmudUEp9GbgO+LhLfTFsvyXb4Gz4qDmGovC493k9VtGWIM1TcKeOxE0mat9wnL5wnF7b\nR8fy1OkdjtE3nLBM2WK8fHSIvuE4/ZPw0skyY6vwUzuKKVuttd6SMWjLmLTJ1JUguIsrAqC1vstx\n+ChwiRv9yGLrZ7PdLcEcl3iRk5Dfy6K6ChbVTW6kYWNHS/VHEvSF42nDtf5IwmHEFqd/2Hjr9Efi\nHOqN8HxkIO23M1E+j9ejjENn0OHcaRmzpY3XAj6qgl6HCZtt0pZpqwr4CPnFiE0QpkoxrAG8H/jV\nWCeVUtcA1wAsW7Ysf73oO9bfftz2MscZLTXZWgFOtNYMxYybpy0IaefOSILBYwzazHH7QITdncZH\nZyCSIDpJzxyPIi0GaTM2y6WzIuClKmDcO6ssF89KhxlbVdqczZc2YrMN22SUIpQzeRMApdQWYOEo\np67XWt9mXXM9kAB+Ptb7aK1vAm4C2LRpU/5yxOuaoO/A6O3ClFFKpe2VZ0I8mSIcTTJomatlXDot\nQ7aY05gtY8QWjiUZjCboGIgQjiWz2qdqNVDhcOVMb/22e6eHSmsEEvI7XDwtU7bgiOOQ35i4Bf2e\ntJmb06hNxEYoJHkTAK11y3jnlVLvBd4GbNbjmVUUis03jFrkhM03uNcnAb/XQ12lh7rK3HjmaK2J\nJVOWM2eSYUsUhqLGXC0cSxKOJRhO79vtCYZjKYbjiXRb/3Ccjv5k+lrbqC2enP6vs+3eGfLbopBx\n9AxYrp6BEcdBh5un85zt5hl0unr6jEmb3ZZ2/rT8Y/xeD36P5SXjNaZtk7UbEEoPt6KAzgP+BXiD\n1jo82dd1D8W4c2db2pphbnWQqkCOhuhTLHIilCZKKeum6aW+Mj+fkUimiCRShGMJonFjuhaJp9Lu\nnZG4cfm0953unVF7a523TdvMuVTa9dM2cYs6TN1iydSoyYgzxXbrdJqM+WyHT0/G5dNnuX/6PCrr\n2KuyHT6zfiw/G6+HzL7tf2Oft1xA7fMq7Ztje+s4vXQynjwep+Nnet/ZlvHksbHvJZO5o9j/1COf\nX/UI90/bgwiHQ6jO2s/4G6UcPkaZY00ylfEzyvgkmfbUMW2Z40RKs2ashFdAufHwrZTaDQSBo1bT\no1rrD0/0uuCiNXrRX/9ndpvPY2XYBo/Jss1k3maycCsDxbDsIQi5R1t/8E5Xz2ja1dO4eMaSybSD\nZ8x5Lu3kmXHzTKS0dWy5eSaNe2ciZa5NWBbEiVTKsiLWxFPGkjiezFgWZ9w+U2nL40QqRSqF1Za5\niaWclscOkzfBkLG3dgjlaEZxDpHduLSer19+ypNa600j388VAZguJ59yqv75H+6la9AkTWWybc1+\n95DZ7xyMjhn/HvJ7mFuVbcMwpyqTaZvOuLWyb+uk+LgguI4tDHrEU7Ldph1PzKnUsU/ZKUtE7Kfz\njBOodjzJ25927D1R62PrAhiOdQWFY10/wYxi7NGHxzEicTqBknYIzbxH2iVUMe3ZDqXUqAJQUo/D\nfq+H9UvqJrxOa004lkwLg51hm864HYrRY2Xa7ukcpHsoRjg2egFyj8Jk11b6aah0WDFUmUxbk3Vr\nLBnsbV2ln6BPHEMFIVd4PArPpCZmhKlQUgIwWZRS6SLey+ZObqJ3pDVDb9js9wzF6E7vxznQHWb7\nwV56huLj1oqt8HvT1gv1lh1D2ophDHuG2go/NUGfLLoJglAQylIApsNUrRnsUUbvcJwey6vHtmLo\nH7ZsGqzM295wjL1dg+nj8ewZPApqQhlRcFozjLRjyBxnMm9l5CEIwmQRAZgmzlHGVBOlIvGkJQax\ntCXDyJ9+x/6RvmH6hhP0D48/6gCzxlETcoqC2c9usywZgv60NUNthdlWB8XjRxBmCyIALhDye1lY\n52Vh3eScQ53Y8efGr8dYMYy0aRhwtPcNxznYHabfysIdb/RhUxXwUm1ZNFRb1gy27UKNbckQclo1\nOK/zpoWxKiBiIgjFjAhAiRGyMkvn105dPMAIiNOaYaRNg23NMBRNMGDtD0bidA2YGgb2z2RD8yr8\nXqosUagMGIEw24xVg3Nb6bBsqPDb1g3Zlg0Br2TMCkIuEAGYZdgC0lgTnPZ7aK2JxFNpW4asbSyZ\ntmpI2zbEkoSjCQajJqO2NxzjUO8wYcf1iSnEens9ikq/7d+TLQ7ZPj4+KgK2VYNt5eCx2r2EfJ70\na+x/F7vd5/VM+99HEEoFEQBhyiilzI0zMDMhcRJLpBiKJgjHjT2D0+fH2DAYu4XhuBGMcCzJcCxj\nwzAcN4LTORDNtFm2DtPJI/J7leXZ4/DwSfv3OHx+RvH1CVm+PkG/bePgdVg3ZGwagg5rB9vGIeAz\nGbMywhEKgQiAUBSYm1+AOTl+X61NButwPNvrJxI3tgvD8YyIRC2rBvvaYes4OsK6YSiWoHsoRSSR\nJJq2eDB2DVMZyYyFUmT7+lgePoER3j1+n8ry+DHnVdrjx+81Ng3G38fez1g6pK/xZLY+x9bvtHZw\nWEB40zYPps22irAzT0XASoeSEoCXj4a57rfbqa901MWtDDCnyk7CksxdIRulFAGfIuDzUFeRG0O5\n8UhY1gpOn59YItvTJzaKh08s6zpHWzJF3N6m2zTxhPmMgUgibelgWzjEk5pYIpm2apgociwfOO0J\nbA8fWyQ8KnvrzHy12z2WN5CdKev1jPT8UWmPH49JoM1qyz7OtMEIPyDG8P+ZihlQ9m46u/iYbGNH\nFrPGZC7bWcya7AznjB/Qsd5ATt8gp2VGlo1Geh9OXVY/5lcoKQGIJpJs2dVBz1BszCctpTAJVo4M\nXbs27hxLNEarm1sj9XGFHODzmvWDyoDbPclgG4jFLTFIOHx+bC8f4wWkiVu+PmmvH/vYuj6lddr3\nx/b5SbenNMmkZUKWzJiTOf2AsgzL7H3HjS3puLk5b2yZmx0kUynHNYC1tQ3YUjpj+5By3FxNY/ZN\nOdOsHf9eo/0bjm4FkWUk51ANp7goczLLnM5jvdBp92C/xuNxWD9YVhL2SMu+1mvZSjg9gZQtug6v\nII9Hsbqxih+M8btRUl5AmzZt0tu2bUtXq+oZspOvTNJVT9hk7vY4ErDsc71hE+EyFnZ93HpHgtVo\nP7UVvuyErEo/1QHJ3hUEoXgpCy8gG2e1qslaPYApLtIbjtM3bAuGEYm+4Xg6MavXSszqCcfYb9fH\nHR6/vKFSpj5uXaURhVorscquh2vvO+vkZs6bkEiJOhEEodCUpABMF7/XQ2NNcMqRK6mUZjCWyCRg\nDSfSwuBMwrLPD0QSvHw0nG4fb+RhUxnwZhVMt7d2Fq9dK9fO1rXbq9M1dX1SUUoQhCkxqwRgung8\nKv1k3zSNMJVEMpVOsLIzeAcswbATsOyErP7hBANRMzKxM3gHIvFJ1cb1e5UjI9dPtVU4vTrk2A/6\nqQoasbEzdu2M3mqr2HpVUMREEGYDIgAFwOf1WJbSAZZO8z1iCVtEMhm8Jis3bgqsR+1C65ks3sFI\ngq7BGPu6hhiMJhmMxonEJxcR4vUok6Eb8Dkyea1i68FM4pVdhL3S76VyRDavs/C6OS9ZvIJQTIgA\nlAgBn4cGnylSMxMS4S6tgAAACodJREFUyRRD0SQD0Xi6cHomc9fEyDuLrNvF18Mxk7XbPhAh3GVe\nZ+rqTi3RypnF68zctbN1K53F10cUYE+/zp9JxnJeYydticgIwuQQAZhl+PJQZD2aSFmF1R3F1K1s\n3aERWbzhWMKRpZtddL1rMJpOwLKzfKdTYN2jMpYXoXSBdWcmrydzPp29680UY/d50tm9IwuzZ7J7\ns/ft5CwRHqGUEAEQZoRSKn0znenoZDTiyVQmKzeWSouInWjlFAs78Wo4lp3JazJ2M5m8A9ZUmLNY\nezSeyknCVMbiIdveYaTlg9+bEQ5n9m7Q50nv2+eMPURm328VYff7PPg9zozfTFavM9vX53Fk+XoV\nfo9HwpYFoMQEoDcc574XOqi3Mn7tYigSQlm+2DfDmlD+s3hTKW1l8RohsbfReCaL1xaNkdm90ZHZ\nvXZGbyKVle0bTSQJhxPErGxdO3s3NiLbN9910D1WcpFt6eD3Zls6jDx2Wj3YFhB2dq9diNx53iQr\nMWbmr1I4ippnEp+cSVB28fNM9i+gso8VKivJyuPJrsM7MisYSCdjOev5OrOEM2fGJ1NLWI84NkeO\n3LOsWsTORLVMmzM5LZMlbDKBTbudCKedSXIjMoOdGcD28Xih8iUlAAd6wrz3f544pr0m6Dsmecuu\nopVVOStdTcuX3g/5pYKWYPB4FCGPtyh+J5IpbSwerExcWxjiVobusfYP2Vm99rHz9cYaIuXI4M0+\nts8nU5p4ys7iTTnOZTJ6h5PGaiLlyAS2b1CJVIpUihHZvhmLgpQmq72EclFLgkw2sBG+01c2jHlt\nSWUCn7TxVP3j27akE7fGq6Jlx+sPx0cv9m4T9HksMciIwsiErXQ1LcexHatfFfDKvK8gzACts+0f\ntCbL48Y+n0xpY9lgPTE7r4cRXjpkP4mn0k/jlhfPiOPs/ozSR3SW1YPNyD/9zCgj40PkPPZYIw+n\nVYTT8gHIsnxQjnPeESMipTK+SLaf0lj3orLIBA76PJy2fGw1G41oIplJ3BpROat/lKpaveEYr3SH\nGbCqaU20COlRWIlYmazempAde28StpxVtWrsGH3HcVXQhEnKvKwwG1FK4VXgndTEi5BLSkoApkPQ\n56WxZnq+9XaEiy0SA5F4OjHLPrYTvExcvonRPzoUY//RcLptMrH3SpGVjGUnb1UFsksyVlsx+PZ+\nZTC7ypYdry+JXIIgTETZC8BMcEa4zK+Z/vvYxU5ssbBj7/sjcYasBK3BaJJBSzBMm7nu6KAREjsm\nf7JhkT47kStdn9cutWiXWbTa0gle2ZW1qoI+q5yjL11hq9LvlQV3QSgjRAAKQLrYSQ7CJKOJ5DGJ\nWoPWsS0sdky+M9HLLoRyuHeYcMy8ZthK7prqd6l0JGVVBnzp0oyVo5RhrMxK1PJkkrxGlGGscMTp\ny1SYIBQGVwRAKfU54EIgBXQA79VaH3ajL6WGSUrKXcx9KqWJWKJiJ2nZ26GoKbU4FHUkbsUThKOZ\nMox2MtfRwRgHrPh8Z6z+dAj4POl6vSGr7GK6BKMzuWvENrsko1Wq0VmW0Z8pw2gnewW9pj3gldh4\nYfbh1gjgRq31vwEopa4FbgA+7FJfZjUej7J8e3xAbur72tjiYouCnaBll2a0k7Wi9vm4fT5TZtE+\nts/1DcfpcCR9ObczjZ13lmHMrt2bncQV9JkSjEFHopczwSt9zYi2gJWYZbdlyjlmSjv6RpR0tBO5\nZD1HyAeuCIDWut9xWEV2RTWhTMiIS/4/S1sVqUYThpHJXJF40lGW0Wq3MoHt65yJXNFEMp3INRRO\nHJPc5SzpmIuawKPhFAM7y9fnyRSRz2QCZ2cFZ9Xzddb19U6U9OWxErnM9emSjio74Ss7wStTrSpd\nlcp53hGbbieKmQigTInHTGJYJkQyXQLSk1360a6qlQ6ZdJwrBcHU6fKQ2aUi7TBVZzUzrUcPl7Xz\nL5z7zmuSKW093I2Oa2sASqkvAO8B+oA3TuY1Ow71sbtjgOaZrMgKZYlSKn2TdPO3w84mjjkSt+IJ\nnZXlG0/aWb+mtm/WcTKT4BVLmPKN8XT5xmOTvOxkrnhSk0xlSj1G4ymGUsmsko72dSmtM9db75NK\nWQlcZfYoli6zmJUJbBqV45p0+4g2m5OW1LHjUF9WmzNfYGRJSbsGsH3OWR+40P/GZ7iRCKaU2gIs\nHOXU9Vrr2xzXXQeEtNafGuN9rgGusQ7XAi/kuq+jMA/oKsDnFIpy+z5Qft9Jvk/xU8rfabnWunFk\no+uZwEqpZcAdWuv1rnbEgVJq22hZc6VKuX0fKL/vJN+n+CnH7+RKULdSao3j8ELgeTf6IQiCMJtx\naw3g35VSazFhoC8jEUCCIAgFx60ooHe68blT4Ca3O5Bjyu37QPl9J/k+xU/ZfSfX1wAEQRAEdxBj\nF0EQhFmKCMAYKKU+rZQ6pJR6xvp5i9t9ygVKqY8ppbRSap7bfZkpSqnPKaW2W/9/7lJKLXa7TzNB\nKXWjUup56zvdqpSqd7tPM0EpdalSaqdSKqWUKtnoGaXUeUqpF5RSu5VS/+p2f3KJCMD4fF1rvdH6\nucPtzswUpdRS4M3AK273JUfcqLXeoLXeCPwBYylSytwNrNdabwBeBK5zuT8zpRW4GHjA7Y5MF6WU\nF/gOcD6wDrhSKbXO3V7lDhGA2cXXgX+hTKw3ys1SRGt9l9Y6YR0+CjS52Z+ZorXepbUuROJmPjkd\n2K213qu1jgG/xISulwUiAOPzUWs4/iOl1By3OzMTlFIXAoe01s+63ZdcopT6glLqAPBXlP4IwMn7\ngT+53QmBJcABx/FBq60smNX1AMazqwD+C/gc5qnyc8BXMX+URcsE3+cTmOmfkmIiSxGt9fXA9Zal\nyEeBUS1FioXJWKQopa4HEsDPC9m36TBZyxehOJnVAqC1bpnMdUqpH2DmmIuasb6PUuokYCXwrOWS\n2AQ8pZQ6XWvdVsAuTpnJ/j/C3CzvoMgFYKLvo5R6L/A2YLMugRjtKfz/KVUOAUsdx01WW1kgU0Bj\noJRa5Di8CLOgVZJorXdoredrrVdorVdghrGnFvvNfyLKzVJEKXUeZo3m7VrrsNv9EQB4AlijlFqp\nlAoAVwC/d7lPOWNWjwAm4CtKqY2YKaD9wIfc7Y4wCuVmKfJtTFWeu62R2qNa65L9Tkqpi4BvAY3A\nH5VSz2itz3W5W1NCa51QSn0UuBPwAj/SWu90uVs5QzKBBUEQZikyBSQIgjBLEQEQBEGYpYgACIIg\nzFJEAARBEGYpIgCCIAizFBEAQcgRSqk/K6V6lVJFnzQoCCACIAi55EbgKrc7IQiTRQRAEKaIUupV\nlklgSClVZXner9dabwUG3O6fIEwWyQQWhCmitX5CKfV74PNABfC/WuuStQoRZi8iAIIwPT6L8YmJ\nANe63BdBmBYyBSQI02MuUA3UACGX+yII00IEQBCmx/eBf8PYUH/Z5b4IwrSQKSBBmCJKqfcAca31\nL6yasQ8rpc4BPgMcD1QrpQ4CV2ut73Szr4IwHuIGKgiCMEuRKSBBEIRZigiAIAjCLEUEQBAEYZYi\nAiAIgjBLEQEQBEGYpYgACIIgzFJEAARBEGYpIgCCIAizlP8P2RdStcl0OG8AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EmqEzUOdhEDP",
        "colab_type": "text"
      },
      "source": [
        "####7.4.2.1. 指数加权移动平均\n",
        "为了从数学上理解动量法，让我们先解释一下指数加权移动平均（exponentially weighted moving average）。给定超参数$0 \\leq \\gamma < 1$，当前时间步 $t$ 的变量 $y_t$ 是上一时间步 $t-1$ 的变量 $y_{t-1} $和当前时间步另一变量 $x_t $的线性组合：\n",
        "$$\n",
        "y_t = \\gamma y_{t-1} + (1-\\gamma) x_t.\n",
        "$$\n",
        "我们可以对 $y_t$ 展开：\n",
        "$$\n",
        "\\begin{split}\\begin{aligned}\n",
        "y_t  &= (1-\\gamma) x_t + \\gamma y_{t-1}\\\\\n",
        "         &= (1-\\gamma)x_t + (1-\\gamma) \\cdot \\gamma x_{t-1} + \\gamma^2y_{t-2}\\\\\n",
        "         &= (1-\\gamma)x_t + (1-\\gamma) \\cdot \\gamma x_{t-1} + (1-\\gamma) \\cdot \\gamma^2x_{t-2} + \\gamma^3y_{t-3}\\\\\n",
        "         &\\ldots\n",
        "\\end{aligned}\\end{split}\n",
        "$$\n",
        "令$n = 1/(1-\\gamma)$，那么$\\left(1-1/n\\right)^n = \\gamma^{1/(1-\\gamma)}$。因为$$\\lim_{n \\rightarrow \\infty}  \\left(1-\\frac{1}{n}\\right)^n = \\exp(-1) \\approx 0.3679,$$\n",
        "\n",
        "所以当$\\gamma \\rightarrow 1$时，$\\gamma^{1/(1-\\gamma)}=\\exp(-1)$，如$0.95^{20} \\approx \\exp(-1)$。如果把 exp(−1) 当作一个比较小的数，我们可以在近似中忽略所有含$\\gamma^{1/(1-\\gamma)}$和比$\\gamma^{1/(1-\\gamma)}$更高阶的系数的项。例如，当 $\\gamma=0.95$时，\n",
        "$$\n",
        "y_t \\approx 0.05 \\sum_{i=0}^{19} 0.95^i x_{t-i}.\n",
        "$$\n",
        "因此，在实际中，我们常常将 $y_t $看作是对最近 $1/(1-\\gamma)$个时间步的 $x_t$ 值的加权平均。例如，当$\\gamma = 0.95$时， $y_t$ 可以被看作对最近20个时间步的 $x_t$ 值的加权平均；当$\\gamma = 0.9$时， $y_t$ 可以看作是对最近10个时间步的 $x_t$ 值的加权平均。而且，离当前时间步 $t$ 越近的 $x_t$ 值获得的权重越大（越接近1）。\n",
        "\n",
        "####7.4.2.2. 由指数加权移动平均理解动量法\n",
        "现在，我们对动量法的速度变量做变形：\n",
        "$$\n",
        "\\boldsymbol{v}_t \\leftarrow \\gamma \\boldsymbol{v}_{t-1} + (1 - \\gamma) \\left(\\frac{\\eta_t}{1 - \\gamma} \\boldsymbol{g}_t\\right).\n",
        "$$\n",
        "由指数加权移动平均的形式可得，速度变量 $v_t$ 实际上对序列$\\{\\eta_{t-i}\\boldsymbol{g}_{t-i} /(1-\\gamma):i=0,\\ldots,1/(1-\\gamma)-1\\}$做了指数加权移动平均。换句话说，相比于小批量随机梯度下降，动量法在每个时间步的自变量更新量近似于将前者对应的最近 $1/(1-\\gamma)$个时间步的更新量做了指数加权移动平均后再除以 $1-\\gamma$\n",
        "。所以，在动量法中，自变量在各个方向上的移动幅度不仅取决当前梯度，还取决于过去的各个梯度在各个方向上是否一致。在本节之前示例的优化问题中，所有梯度在水平方向上为正（向右），而在竖直方向上时正（向上）时负（向下）。这样，我们就可以使用较大的学习率，从而使自变量向最优解更快移动。\n",
        "\n",
        "###7.4.3. 从零开始实现\n",
        "相对于小批量随机梯度下降，动量法需要对每一个自变量维护一个同它一样形状的速度变量，且超参数里多了动量超参数。实现中，我们将速度变量用更广义的状态变量states表示。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-InNFn3fD8su",
        "colab_type": "code",
        "outputId": "0e87eb0d-1154-40c4-aae5-0b301773dbcf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 127
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "def get_data_ch7():\n",
        "  data=np.genfromtxt('/content/drive/My Drive/data/d2l-zh-tensoflow/airfoil_self_noise.dat',delimiter='\\t')\n",
        "  data=(data-data.mean(axis=0))/data.std(axis=0)\n",
        "  return tf.constant(data[:1500,:-1]),tf.constant(data[:1500,-1])\n",
        "\n",
        "features,labels=get_data_ch7()  "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O_tHnfiuEizM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def init_momentum_states():\n",
        "  v_w=tf.Variable(tf.zeros(shape=(features.shape[1],1),dtype=tf.double))\n",
        "  v_b=tf.Variable(tf.zeros(1,dtype=tf.double))\n",
        "  return (v_w,v_b)\n",
        "\n",
        "def sgd_momentum(params,states,hyperparams,loss,t):\n",
        "  for p,v in zip(params,states):\n",
        "    dl_dp=t.gradient(loss,p)\n",
        "    v.assign(hyperparams['momentum']*v+hyperparams['lr']*dl_dp)\n",
        "    p.assign_sub(v)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CKi-HrUdHZzc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_ch7(trainer_fn,states,hyperparams,features,labels,batch_size=10,num_epochs=2):\n",
        "  net,loss=linreg,squared_loss\n",
        "  w=tf.Variable(tf.random.normal(stddev=0.01,shape=(features.shape[1],1),dtype=tf.double))\n",
        "  b=tf.Variable(tf.zeros(1,dtype=tf.double))\n",
        "\n",
        "  def eval_loss():\n",
        "    return tf.reduce_mean(loss(net(features,w,b),labels)).numpy()\n",
        "\n",
        "  ls=[eval_loss()]\n",
        "\n",
        "  data_iter=Dataset.from_tensor_slices((features,labels)).shuffle(buffer_size=1000).batch(batch_size)\n",
        "\n",
        "  for _ in range(num_epochs):\n",
        "    start=time.time()\n",
        "    for batch_i,(X,y) in enumerate(data_iter):\n",
        "      with tf.GradientTape(persistent=True) as t:\n",
        "        t.watch([w,b])\n",
        "        l=tf.reduce_mean(loss(net(X,w,b),y)) #使用平均损失\n",
        "\n",
        "      trainer_fn([w,b],states,hyperparams,l,t) #迭代模型参数\n",
        "\n",
        "      if(batch_i+1)*batch_size %100==0:\n",
        "        ls.append(eval_loss()) #每100个样本记录下当前训练误差\n",
        "  # 打印结果和作图\n",
        "  print('loss: %f, %f sec per epoch' % (ls[-1], time.time() - start))\n",
        "  set_figsize()\n",
        "  plt.plot(np.linspace(0, num_epochs, len(ls)), ls)\n",
        "  plt.xlabel('epoch')\n",
        "  plt.ylabel('loss')  \n",
        "\n",
        "def use_svg_display():\n",
        "  #用矢量图显示\n",
        "  display.set_matplotlib_formats('svg')\n",
        "def set_figsize(figsize=(3.5,2.5)):\n",
        "  use_svg_display()\n",
        "  #设置图的尺寸\n",
        "  plt.rcParams['figure.figsize']=figsize\n",
        "  \n",
        "  \n",
        "def linreg(X,w,b):\n",
        "  return tf.matmul(X,w)+b \n",
        "  \n",
        "def squared_loss(y_hat,y):\n",
        "  return (y_hat-tf.reshape(y,shape=y_hat.shape))**2/2    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NeHKy-Rkja4Z",
        "colab_type": "text"
      },
      "source": [
        "我们先将动量超参数momentum设0.5，这时可以看成是特殊的小批量随机梯度下降：其小批量随机梯度为最近2个时间步的2倍小批量梯度的加权平均。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gFSwVbEEHniE",
        "colab_type": "code",
        "outputId": "49b39272-0157-41d7-956a-7e802df509a9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "train_ch7(sgd_momentum,init_momentum_states(),{'lr':0.02,'momentum':0.5},features,labels)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loss: 0.248673, 0.431266 sec per epoch\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 252x180 with 1 Axes>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"182.935714pt\" version=\"1.1\" viewBox=\"0 0 252.64375 182.935714\" width=\"252.64375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 182.935714 \nL 252.64375 182.935714 \nL 252.64375 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 50.14375 145.379464 \nL 245.44375 145.379464 \nL 245.44375 9.479464 \nL 50.14375 9.479464 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m87258a709c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"59.021023\" xlink:href=\"#m87258a709c\" y=\"145.379464\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0.0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n       <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n      </defs>\n      <g transform=\"translate(51.06946 159.977902)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"103.407386\" xlink:href=\"#m87258a709c\" y=\"145.379464\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0.5 -->\n      <defs>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g transform=\"translate(95.455824 159.977902)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"147.79375\" xlink:href=\"#m87258a709c\" y=\"145.379464\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 1.0 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n      </defs>\n      <g transform=\"translate(139.842187 159.977902)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"192.180114\" xlink:href=\"#m87258a709c\" y=\"145.379464\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 1.5 -->\n      <g transform=\"translate(184.228551 159.977902)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.566477\" xlink:href=\"#m87258a709c\" y=\"145.379464\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 2.0 -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g transform=\"translate(228.614915 159.977902)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_6\">\n     <!-- epoch -->\n     <defs>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-104\"/>\n     </defs>\n     <g transform=\"translate(132.565625 173.656027)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_6\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"m5b1d51befa\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m5b1d51befa\" y=\"135.631194\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 0.25 -->\n      <g transform=\"translate(20.878125 139.430413)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_7\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m5b1d51befa\" y=\"110.704799\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.30 -->\n      <defs>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g transform=\"translate(20.878125 114.504018)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m5b1d51befa\" y=\"85.778404\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.35 -->\n      <g transform=\"translate(20.878125 89.577623)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_9\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m5b1d51befa\" y=\"60.852009\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.40 -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g transform=\"translate(20.878125 64.651228)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m5b1d51befa\" y=\"35.925614\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.45 -->\n      <g transform=\"translate(20.878125 39.724833)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_11\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m5b1d51befa\" y=\"10.999219\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 0.50 -->\n      <g transform=\"translate(20.878125 14.798438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- loss -->\n     <defs>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g transform=\"translate(14.798437 87.087277)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#p3a4e33b162)\" d=\"M 59.021023 15.656737 \nL 64.939205 67.690453 \nL 70.857386 96.137897 \nL 76.775568 112.027619 \nL 82.69375 120.45703 \nL 88.611932 126.963048 \nL 94.530114 130.530657 \nL 100.448295 134.847155 \nL 106.366477 136.42357 \nL 112.284659 136.584064 \nL 118.202841 134.097771 \nL 124.121023 132.155056 \nL 130.039205 138.472949 \nL 135.957386 138.626725 \nL 141.875568 137.780742 \nL 147.79375 135.667618 \nL 153.711932 137.157478 \nL 159.630114 136.387139 \nL 165.548295 137.142117 \nL 171.466477 138.763698 \nL 177.384659 138.561403 \nL 183.302841 138.037273 \nL 189.221023 138.707597 \nL 195.139205 138.921432 \nL 201.057386 138.234362 \nL 206.975568 135.583179 \nL 212.89375 133.775032 \nL 218.811932 139.162751 \nL 224.730114 139.202192 \nL 230.648295 138.295046 \nL 236.566477 136.292609 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 50.14375 145.379464 \nL 50.14375 9.479464 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 245.44375 145.379464 \nL 245.44375 9.479464 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 50.14375 145.379464 \nL 245.44375 145.379464 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 50.14375 9.479464 \nL 245.44375 9.479464 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p3a4e33b162\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"50.14375\" y=\"9.479464\"/>\n  </clipPath>\n </defs>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vtOrPRkLjeaQ",
        "colab_type": "text"
      },
      "source": [
        "将动量超参数momentum增大到0.9，这时依然可以看成是特殊的小批量随机梯度下降：其小批量随机梯度为最近10个时间步的10倍小批量梯度的加权平均。我们先保持学习率0.02不变。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PfV-fLFTIFKF",
        "colab_type": "code",
        "outputId": "2fa1e52c-c8d9-4aca-a2b9-ebe373ca0b06",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        }
      },
      "source": [
        "train_ch7(sgd_momentum,init_momentum_states(),{'lr':0.02,'momentum':0.9},features,labels)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loss: 0.272450, 0.440948 sec per epoch\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 252x180 with 1 Axes>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 252.64375 180.65625\" width=\"252.64375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 180.65625 \nL 252.64375 180.65625 \nL 252.64375 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 50.14375 143.1 \nL 245.44375 143.1 \nL 245.44375 7.2 \nL 50.14375 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m6236fa3f58\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"59.021023\" xlink:href=\"#m6236fa3f58\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0.0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n       <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n      </defs>\n      <g transform=\"translate(51.06946 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"103.407386\" xlink:href=\"#m6236fa3f58\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0.5 -->\n      <defs>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g transform=\"translate(95.455824 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"147.79375\" xlink:href=\"#m6236fa3f58\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 1.0 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n      </defs>\n      <g transform=\"translate(139.842187 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"192.180114\" xlink:href=\"#m6236fa3f58\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 1.5 -->\n      <g transform=\"translate(184.228551 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.566477\" xlink:href=\"#m6236fa3f58\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 2.0 -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g transform=\"translate(228.614915 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_6\">\n     <!-- epoch -->\n     <defs>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-104\"/>\n     </defs>\n     <g transform=\"translate(132.565625 171.376563)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_6\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"m100c689efc\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m100c689efc\" y=\"135.227386\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 0.25 -->\n      <g transform=\"translate(20.878125 139.026604)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_7\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m100c689efc\" y=\"111.790758\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.30 -->\n      <defs>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g transform=\"translate(20.878125 115.589977)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m100c689efc\" y=\"88.354131\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.35 -->\n      <g transform=\"translate(20.878125 92.15335)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_9\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m100c689efc\" y=\"64.917503\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.40 -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g transform=\"translate(20.878125 68.716722)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m100c689efc\" y=\"41.480876\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.45 -->\n      <g transform=\"translate(20.878125 45.280095)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_11\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m100c689efc\" y=\"18.044248\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 0.50 -->\n      <g transform=\"translate(20.878125 21.843467)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- loss -->\n     <defs>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#pb9e15f305c)\" d=\"M 59.021023 13.377273 \nL 64.939205 101.589925 \nL 70.857386 127.841901 \nL 76.775568 128.394374 \nL 82.69375 124.737221 \nL 88.611932 132.526003 \nL 94.530114 131.694122 \nL 100.448295 136.922727 \nL 106.366477 125.505949 \nL 112.284659 134.710707 \nL 118.202841 124.333557 \nL 124.121023 129.3426 \nL 130.039205 131.378323 \nL 135.957386 127.761078 \nL 141.875568 136.683581 \nL 147.79375 124.660956 \nL 153.711932 132.988234 \nL 159.630114 130.242485 \nL 165.548295 134.599689 \nL 171.466477 131.543998 \nL 177.384659 135.945897 \nL 183.302841 133.698101 \nL 189.221023 135.863126 \nL 195.139205 124.413373 \nL 201.057386 134.496261 \nL 206.975568 123.635697 \nL 212.89375 129.490171 \nL 218.811932 131.336757 \nL 224.730114 127.646666 \nL 230.648295 136.6346 \nL 236.566477 124.704438 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 50.14375 143.1 \nL 50.14375 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 245.44375 143.1 \nL 245.44375 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 50.14375 143.1 \nL 245.44375 143.1 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 50.14375 7.2 \nL 245.44375 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pb9e15f305c\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"50.14375\" y=\"7.2\"/>\n  </clipPath>\n </defs>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UjpGYWJdjqYW",
        "colab_type": "text"
      },
      "source": [
        "可见目标函数值在后期迭代过程中的变化不够平滑。直觉上，10倍小批量梯度比2倍小批量梯度大了5倍，我们可以试着将学习率减小到原来的1/5。此时目标函数值在下降了一段时间后变化更加平滑。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rXsa1D3uIj2N",
        "colab_type": "code",
        "outputId": "3b9ed48b-ad15-45de-e9ee-55cce28d81c9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "train_ch7(sgd_momentum,init_momentum_states(),{'lr':0.004,'momentum':0.9},features,labels)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loss: 0.242490, 0.432922 sec per epoch\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 252x180 with 1 Axes>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"182.7642pt\" version=\"1.1\" viewBox=\"0 0 252.64375 182.7642\" width=\"252.64375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 182.7642 \nL 252.64375 182.7642 \nL 252.64375 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 50.14375 145.20795 \nL 245.44375 145.20795 \nL 245.44375 9.30795 \nL 50.14375 9.30795 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m352aa7a68a\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"59.021023\" xlink:href=\"#m352aa7a68a\" y=\"145.20795\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0.0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n       <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n      </defs>\n      <g transform=\"translate(51.06946 159.806388)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"103.407386\" xlink:href=\"#m352aa7a68a\" y=\"145.20795\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0.5 -->\n      <defs>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g transform=\"translate(95.455824 159.806388)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"147.79375\" xlink:href=\"#m352aa7a68a\" y=\"145.20795\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 1.0 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n      </defs>\n      <g transform=\"translate(139.842187 159.806388)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"192.180114\" xlink:href=\"#m352aa7a68a\" y=\"145.20795\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 1.5 -->\n      <g transform=\"translate(184.228551 159.806388)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.566477\" xlink:href=\"#m352aa7a68a\" y=\"145.20795\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 2.0 -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g transform=\"translate(228.614915 159.806388)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_6\">\n     <!-- epoch -->\n     <defs>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-104\"/>\n     </defs>\n     <g transform=\"translate(132.565625 173.484513)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_6\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"m265ebcff15\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m265ebcff15\" y=\"135.296778\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 0.25 -->\n      <g transform=\"translate(20.878125 139.095997)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_7\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m265ebcff15\" y=\"110.437266\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.30 -->\n      <defs>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g transform=\"translate(20.878125 114.236485)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m265ebcff15\" y=\"85.577754\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.35 -->\n      <g transform=\"translate(20.878125 89.376973)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_9\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m265ebcff15\" y=\"60.718243\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.40 -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g transform=\"translate(20.878125 64.517461)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m265ebcff15\" y=\"35.858731\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.45 -->\n      <g transform=\"translate(20.878125 39.657949)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_11\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m265ebcff15\" y=\"10.999219\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 0.50 -->\n      <g transform=\"translate(20.878125 14.798437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- loss -->\n     <defs>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g transform=\"translate(14.798437 86.915763)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#pa9023cf2a9)\" d=\"M 59.021023 15.485223 \nL 64.939205 39.28163 \nL 70.857386 75.59011 \nL 76.775568 106.041738 \nL 82.69375 122.816452 \nL 88.611932 129.483175 \nL 94.530114 132.667386 \nL 100.448295 135.752226 \nL 106.366477 137.745931 \nL 112.284659 137.639973 \nL 118.202841 136.930158 \nL 124.121023 137.165342 \nL 130.039205 138.11115 \nL 135.957386 137.084802 \nL 141.875568 137.787369 \nL 147.79375 138.62177 \nL 153.711932 138.533228 \nL 159.630114 138.314672 \nL 165.548295 137.980261 \nL 171.466477 137.91042 \nL 177.384659 137.704304 \nL 183.302841 138.191504 \nL 189.221023 138.049866 \nL 195.139205 138.697313 \nL 201.057386 138.640684 \nL 206.975568 137.882616 \nL 212.89375 137.856195 \nL 218.811932 138.594512 \nL 224.730114 137.472328 \nL 230.648295 138.195384 \nL 236.566477 139.030678 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 50.14375 145.20795 \nL 50.14375 9.30795 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 245.44375 145.20795 \nL 245.44375 9.30795 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 50.14375 145.20795 \nL 245.44375 145.20795 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 50.14375 9.30795 \nL 245.44375 9.30795 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pa9023cf2a9\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"50.14375\" y=\"9.30795\"/>\n  </clipPath>\n </defs>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MdeedHqRjvGC",
        "colab_type": "text"
      },
      "source": [
        "###7.4.4. 简洁实现"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A01mQHeiIrDt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_keras_ch7(trainer_name,features,labels,batch_size=10,num_epochs=2):\n",
        "  net=keras.Sequential(keras.layers.Dense(1))\n",
        "  loss=losses.mean_squared_error\n",
        "  data_iter=Dataset.from_tensor_slices((features,labels)).shuffle(buffer_size=1000).batch(batch_size)\n",
        "  net.compile(optimizer=trainer_name,loss=loss)\n",
        "  history=net.fit_generator(data_iter,epochs=num_epochs)\n",
        "  loss_ls=history.history['loss']\n",
        "  set_figsize()\n",
        "  plt.plot(np.linspace(0,num_epochs,len(loss_ls)),loss_ls)\n",
        "  plt.xlabel('epoch')\n",
        "  plt.ylabel('loss')\n",
        "  plt.show()\n",
        "  return history"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ClEK_sPDI6cd",
        "colab_type": "code",
        "outputId": "a475188b-2b1a-4902-d34c-30f958c714bb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 643
        }
      },
      "source": [
        "train_keras_ch7(keras.optimizers.SGD(learning_rate=0.004,momentum=0.9),features,labels,batch_size=10,num_epochs=10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.5574\n",
            "Epoch 2/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.4977\n",
            "Epoch 3/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.4981\n",
            "Epoch 4/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.4981\n",
            "Epoch 5/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.4981\n",
            "Epoch 6/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.4981\n",
            "Epoch 7/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.4981\n",
            "Epoch 8/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.4981\n",
            "Epoch 9/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.4981\n",
            "Epoch 10/10\n",
            "150/150 [==============================] - 1s 5ms/step - loss: 0.4981\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 252x180 with 1 Axes>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"183.641911pt\" version=\"1.1\" viewBox=\"0 0 252.64375 183.641911\" width=\"252.64375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.641911 \nL 252.64375 183.641911 \nL 252.64375 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 50.14375 146.085661 \nL 245.44375 146.085661 \nL 245.44375 10.185661 \nL 50.14375 10.185661 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"mc622e10baf\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"59.021023\" xlink:href=\"#mc622e10baf\" y=\"146.085661\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g transform=\"translate(55.839773 160.684099)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"94.530114\" xlink:href=\"#mc622e10baf\" y=\"146.085661\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 2 -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g transform=\"translate(91.348864 160.684099)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"130.039205\" xlink:href=\"#mc622e10baf\" y=\"146.085661\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 4 -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g transform=\"translate(126.857955 160.684099)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-52\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"165.548295\" xlink:href=\"#mc622e10baf\" y=\"146.085661\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 6 -->\n      <defs>\n       <path d=\"M 33.015625 40.375 \nQ 26.375 40.375 22.484375 35.828125 \nQ 18.609375 31.296875 18.609375 23.390625 \nQ 18.609375 15.53125 22.484375 10.953125 \nQ 26.375 6.390625 33.015625 6.390625 \nQ 39.65625 6.390625 43.53125 10.953125 \nQ 47.40625 15.53125 47.40625 23.390625 \nQ 47.40625 31.296875 43.53125 35.828125 \nQ 39.65625 40.375 33.015625 40.375 \nz\nM 52.59375 71.296875 \nL 52.59375 62.3125 \nQ 48.875 64.0625 45.09375 64.984375 \nQ 41.3125 65.921875 37.59375 65.921875 \nQ 27.828125 65.921875 22.671875 59.328125 \nQ 17.53125 52.734375 16.796875 39.40625 \nQ 19.671875 43.65625 24.015625 45.921875 \nQ 28.375 48.1875 33.59375 48.1875 \nQ 44.578125 48.1875 50.953125 41.515625 \nQ 57.328125 34.859375 57.328125 23.390625 \nQ 57.328125 12.15625 50.6875 5.359375 \nQ 44.046875 -1.421875 33.015625 -1.421875 \nQ 20.359375 -1.421875 13.671875 8.265625 \nQ 6.984375 17.96875 6.984375 36.375 \nQ 6.984375 53.65625 15.1875 63.9375 \nQ 23.390625 74.21875 37.203125 74.21875 \nQ 40.921875 74.21875 44.703125 73.484375 \nQ 48.484375 72.75 52.59375 71.296875 \nz\n\" id=\"DejaVuSans-54\"/>\n      </defs>\n      <g transform=\"translate(162.367045 160.684099)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-54\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"201.057386\" xlink:href=\"#mc622e10baf\" y=\"146.085661\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 8 -->\n      <defs>\n       <path d=\"M 31.78125 34.625 \nQ 24.75 34.625 20.71875 30.859375 \nQ 16.703125 27.09375 16.703125 20.515625 \nQ 16.703125 13.921875 20.71875 10.15625 \nQ 24.75 6.390625 31.78125 6.390625 \nQ 38.8125 6.390625 42.859375 10.171875 \nQ 46.921875 13.96875 46.921875 20.515625 \nQ 46.921875 27.09375 42.890625 30.859375 \nQ 38.875 34.625 31.78125 34.625 \nz\nM 21.921875 38.8125 \nQ 15.578125 40.375 12.03125 44.71875 \nQ 8.5 49.078125 8.5 55.328125 \nQ 8.5 64.0625 14.71875 69.140625 \nQ 20.953125 74.21875 31.78125 74.21875 \nQ 42.671875 74.21875 48.875 69.140625 \nQ 55.078125 64.0625 55.078125 55.328125 \nQ 55.078125 49.078125 51.53125 44.71875 \nQ 48 40.375 41.703125 38.8125 \nQ 48.828125 37.15625 52.796875 32.3125 \nQ 56.78125 27.484375 56.78125 20.515625 \nQ 56.78125 9.90625 50.3125 4.234375 \nQ 43.84375 -1.421875 31.78125 -1.421875 \nQ 19.734375 -1.421875 13.25 4.234375 \nQ 6.78125 9.90625 6.78125 20.515625 \nQ 6.78125 27.484375 10.78125 32.3125 \nQ 14.796875 37.15625 21.921875 38.8125 \nz\nM 18.3125 54.390625 \nQ 18.3125 48.734375 21.84375 45.5625 \nQ 25.390625 42.390625 31.78125 42.390625 \nQ 38.140625 42.390625 41.71875 45.5625 \nQ 45.3125 48.734375 45.3125 54.390625 \nQ 45.3125 60.0625 41.71875 63.234375 \nQ 38.140625 66.40625 31.78125 66.40625 \nQ 25.390625 66.40625 21.84375 63.234375 \nQ 18.3125 60.0625 18.3125 54.390625 \nz\n\" id=\"DejaVuSans-56\"/>\n      </defs>\n      <g transform=\"translate(197.876136 160.684099)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-56\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.566477\" xlink:href=\"#mc622e10baf\" y=\"146.085661\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 10 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n      </defs>\n      <g transform=\"translate(230.203977 160.684099)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epoch -->\n     <defs>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-104\"/>\n     </defs>\n     <g transform=\"translate(132.565625 174.362224)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"m7e8abb34b0\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m7e8abb34b0\" y=\"135.128062\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.50 -->\n      <defs>\n       <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g transform=\"translate(20.878125 138.927281)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m7e8abb34b0\" y=\"93.751781\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.52 -->\n      <g transform=\"translate(20.878125 97.551)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m7e8abb34b0\" y=\"52.3755\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.54 -->\n      <g transform=\"translate(20.878125 56.174719)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-52\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m7e8abb34b0\" y=\"10.999219\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.56 -->\n      <g transform=\"translate(20.878125 14.798437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-54\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- loss -->\n     <defs>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g transform=\"translate(14.798437 87.793474)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_11\">\n    <path clip-path=\"url(#p99a1f83b9e)\" d=\"M 59.021023 16.362934 \nL 78.748295 139.908388 \nL 98.475568 139.160393 \nL 118.202841 139.074101 \nL 137.930114 139.06638 \nL 157.657386 139.065723 \nL 177.384659 139.065669 \nL 197.111932 139.065652 \nL 216.839205 139.065657 \nL 236.566477 139.065657 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 50.14375 146.085661 \nL 50.14375 10.185661 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 245.44375 146.085661 \nL 245.44375 10.185661 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 50.14375 146.085661 \nL 245.44375 146.085661 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 50.14375 10.185661 \nL 245.44375 10.185661 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p99a1f83b9e\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"50.14375\" y=\"10.185661\"/>\n  </clipPath>\n </defs>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f4c07422048>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8FlavWtjJIF1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}