notebooks_en/2_Logistic_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2021 Lorena A. Barba, Pi-Yueh Chuang, Tingyu Wang"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic regression\n",
    "\n",
    "In Lesson 1 of this module, you learned about fitting a line to data (linear regression) using the method of gradient descent to find the model parameters (slope and $y$-intercept). But what if the observational data looks nothing like it has a linear relationship?\n",
    "\n",
    "A major class of problems deals with binary classification, that is, the data belong to one of two categories. \n",
    "Typically, we code the two categories with $0$ and $1$. For example, is an email spam, or not spam? Is a credit-card transaction fraudulent, or legitimate? \n",
    "In these settings, often the data correspond to numbers between $0$ and $1$ that represent some _probability_ (e.g., that the email be spam or not), and one applies a decision boundary to assign each item to a class.\n",
    "\n",
    "One effective way to build a model representing this situation is logistic regression. The output of the model is a number between $0$ and $1$, for inputs in the whole real line. \n",
    "With binary data (zeros and ones), we might imagine that a good model could look like a step function. But this function is not differentiable. Since we want to use gradient descent, we adopt a continuous function that approximates the step function. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic function\n",
    "\n",
    "We can build a model that will output a value between zero and one by making a non-linear transformation of the linear regression. \n",
    "This is achieved with the _logistic function_:\n",
    "\n",
    "$$ \\sigma(z) = \\frac{1}{1+e^{-z}}$$\n",
    "\n",
    "With the data consisting of two arrays, $x, y$, the model is a composition: $z= wx + b$ and $\\sigma(z)$ is the output.\n",
    "The task is finding $w$ and $b$ so that $\\sigma(z)$ is closest to $y$ for all data points.\n",
    "\n",
    "Let's play with this function using SymPy. We'll need NumPy and Matplotlib later, so might as well load all our libraries now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sympy\n",
    "import numpy\n",
    "\n",
    "from matplotlib import pyplot\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{1}{1 + e^{- z}}$"
      ],
      "text/plain": [
       "1/(1 + exp(-z))"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = sympy.Symbol('z', real=True)\n",
    "\n",
    "logistic = 1/(1+ sympy.exp(-z))\n",
    "logistic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAEfCAYAAAAQiIIwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAp+0lEQVR4nO3deVxc9b3/8ddhYNgSCCQhgQyEEBKSkBAMQ4y21XqrZqnGGq821lYt9dK4tPa2P7W/9nd/t/W2apdfWx+Nt7lp3a+aq1aNmsXGaNyySVbNioYkQEjYEgg7M/P9/QFBCBCyAGdg3s/HYx5wzvmeMx/OLG/O9j2WMQYRERF/FWR3ASIiImeioBIREb+moBIREb+moBIREb+moBIREb+moBIREb+moBIREb+moBIREb+moBIREb8WbHcBIhfCsqww4BrgK0ACUA98CqwwxuyyszYR6R2WulCSgcqyrF8A1wLrgC1AKRAGTASuaP39J8aYnTaVKCK9QEElA5ZlWV83xqw4w/Q4IMkYk9ePZYlIL1NQyYBnWdZXgPXGGG+7cTOMMVttLEtEeolOppDB4C3gHcuyRrUb9ze7ihGR3tVTUBk99PD3R2ZmZvgbb7xx2aRJk46uX7/+1LiL7Khlzpw5tq8PPfQYYI8e6aw/GfAsy+Kaa64hLS2Nb37zm+Tk5GBZli21lJeX2/K8IoOZdv3JgHfqOOuECRP48MMPef/999m5Uyf6iQwWPZ1McVabZSL+5vDhwyQlJfX787rdbvLydJKhyDnocfeHtqhkwPrVr35FZWVll9OSkpJ45513ePPNN/u5KhHpbTpGJQPWtGnTuPbaawkLC2PGjBmMHDmShoYG8vPz2b59O1deeSU/+9nP7C5TRC6QgkoGrJdffpmPPvqI3/72t8TFxVFSUkJUVBTf/va3Wbp0KeHh4XaXKCK9QEElA9aWLVs4dOgQzz33HO+++26HafX19QoqkUFCQSUD1qJFi5gzZw4HDhzA7Xa3jTfGYFkWBw4c6HbenJwc3nzzTeLi4vj00087TTfGcO+997Jy5UoiIiJ46qmnmDFjRp/8HSJyZjqZQgasH/7wh+zZs4ecnBwOHDjQ9igoKDhjSAHcfvvtrF69utvpq1atIj8/n/z8fJYuXcqdd97Z2+WLyFnSFpUMeH/5y1/OeZ7LLruMgwcPdjt9+fLl3HrrrViWxaxZszhx4gQlJSXEx8dfQKUyUPh8hiavjyavj2bPFz89PoPXZ/D4DB6vwePzYYyhyWvw+Qxe0zLdZww+A16fwRiD10frOEOww6Kx2YfPtGy5GwNWEHi8Lb8bWn8aQ3iIg9omb0s7wBgIdzqobfS01WoMWFbL8k8NwxfXFoWHOKhr8mLaXW0U6Qympt0yACKcre3MmdsBzJ0WT2rckN5Z2WdBQSXSheLiYhITE9uGXS4XxcXFXQbV0qVLWbp0KQBlZWX9VqO0aPb6OFHXzIm6Jo7XNVPb5KGypomaRg81jR5qW3/WNHoYGhpMQUUdDc3edg8f9c1ekmIj2HWkimZvS9icLj46jJKqhk7jMxOj2V5Y1Wl8dHgwVfWdv+TdY2PIO3T8tGWHUlLV2KntxeNi2VTQ8RIMd3IMeQc7zt/dcwFkuKLZWdSxvq6We1HSMLYdPtFhXNbYGLacVivAhFFDFVQiduvqQvjuumXKzc0lNzcXoMOxMrkwXp/haHUDhZW1FB9voPRkI8eqGyg92UBpdSMjhoTy4Wflnf7jPz0IgoMshoQFE+kMJj0hiur6ZsJCgoiNdBIW7CDc6SAsJIiRQ0LJGhtDiMPC6XAQEmzhdAThDA4ixBFEWEgQFhbBDovgIAtHUBDBDgunw8LCIijIwhFkEWSd+knbcMu4lveQZYHDapnHsmh7BJ0+DgswLeOtliHLatniCrKCOPV2PPWuDGr3/vximoXP+HAEdTzKY4whKMg6bdwXyzg1v/F1bnf6c/UHBZVIF1wuF4WFhW3DRUVFJCQk2FjR4FXb6GH/sZPsO3qSvUdPUnS8js/Laik6Xkez17R9QfsMDA0NZmRUKKOGhjEmJowb3S6GhTuJiQxhWISTmIgQhoWHEBUeQmRoMENCgwkNDrKt70f/4LBl1t6koBLpwvz581m8eDELFy5k06ZNREdH6/hUL2j0ePmkqIpNBZXkl9aw9dBxDlfWtU2PcDrITo5lSkIUc6aOJjEmgqTYCOKHhTE6KozIUH1lBSK96hKQbr75ZtatW0d5eTkul4tf/vKXNDc3Ay2nvc+bN4+VK1eSmppKREQETz75pM0VD0z1TV62HT7OxoJKNhdUsO3wCRo9PgAunziSaWOi+ecsF5NGD2XS6ChcMeFd7mqSwKZOaUV6kTqlhZMNzazdU8qKT0qoa/Lw0WcVBFmQnhDNzHGxzBwXS3ZyLLGRTrtLFf/Q438m2qISkQtW0+hh7Z5jrNhZwrr9ZTR5fIyOCuO6zHju+EoKWWNjiAoLsbtMGaAUVCJy3gor6/jbBwcoqKjj/f1lxA0N5Vszk7gmI54ZSTHajSe9QkElIudsT0k1S977nDd3lhBkwU1Zibz4/Utwj1U4Se9TUInIWTHGsPFAJUve+5z39pcR6XTw3UuT+d5XxhEfrQ6Ape8oqESkRzuLTvDU+oO8srWY4ZFO/tfVE/nOrGSiI3TcSfqegkpEulXf5OUPa/bx+IcFTImP4j++MZUbs1yEhfjJlaASEBRUItKl9Z+V89NXPuFwZR03z0zip3MnER2uLSjpfwoqEemgqr6Zh1fuYdnHhSQPj+CFf5nFJeOH212WBDAFlYi02VxQwT3Pb6OitonvX57Cv145Ubv5xHYKKhEB4NmNh3jw9V18KXUEP7k6jWmuaLtLEgEUVCIBzxjD7/+xj8fe/ZwrJ8fx55tnEO7UVpT4DwWVSABr9vp44O87eWVrMTfPTOQ/rptKsCOo5xlF+pGCSiRA1TR6uOu5rby/v4wfXzWRH/xTaoDft0n8lYJKJACVnmwg56mP2VNykt/ekMFN2Yl2lyTSLQWVSIA5UFbDbU9upvxkE3+71c0Vk+LsLknkjBRUIgHkUEUtD63cQ12jl2W5s5ieOMzukkR6pKASCRDVDc187+k8yk428D/fv4RJo6PsLknkrCioRAKAx+vjB89v42B5Lc9+72KFlAwoCiqRAPDQyr28t7+MhxdMU3dIMuDoggmRQe6FzYd54qMCvvulZG6emWR3OSLnTEElMoht+LyCf3vtUy6fOJKfz5tsdzki50VBJTJIHSyv5c7ntpA8IpI/f+si9TghA5beuSKDUMsZfh8D8PhtbqLCdB8pGbgUVCKD0O/f2kdYiIO/3JLF2OGRdpcjckEUVCKDzLv7SnlmwyGunBynM/xkUFBQiQwitY0e/s+rn5IaN4S7rki1uxyRXqHrqEQGkf/3j/0Un6jn5UWXEBqse0rJ4KAtKpFBYkfhCZ5aX8C3ZyXhTo61uxyRXqOgEhkETt0AceTQUO6fM8nuckR6lXb9iQwCf/3gAHuPnuS/vpOlU9Fl0NEWlcgAd7C8lkffzmd2+ihmp4+2uxyRXqegEhnAjDH8/LVPcDqCePC6qXaXI9InFFQiA9jLW4r46LMKHpg7iVFRYXaXI9InFFQiA1R5TSO/XrmH7OQYvqVe0WUQU1CJDFB/++AAdY1eHl4wjaAgy+5yRPqMgkoC1urVq0lLSyM1NZVHHnmk0/SqqiquvfZapk+fTnp6Ok8++aQNVXZt79Fq/uv9A9x1xXhS44baXY5In1JQSUDyer3cfffdrFq1it27d/PCCy+we/fuDm0ee+wxpkyZwo4dO1i3bh0/+clPaGpqsqnijh59O58hzmBuvzTZ7lJE+pyCSgLS5s2bSU1NJSUlBafTycKFC1m+fHmHNpZlcfLkSYwx1NTUEBsbS3Cw/Zce7jpSxapPj5Lz5XEMi3DaXY5In1NQSUAqLi4mMTGxbdjlclFcXNyhzT333MOePXtISEhg2rRpPProowQF2f+R+dPb+USFBZPz5XF2lyLSL+z/1InYwBjTaZxldTwh4a233iIzM5MjR46wfft27rnnHqqrqzvNt3TpUtxuN263m7Kysj6rGeCToirW7D7Gv3wlhehw9UAhgUFBJQHJ5XJRWFjYNlxUVERCQkKHNk8++SQLFizAsixSU1MZN24ce/fu7bSs3Nxc8vLyyMvLY+TIkX1a9x/f3s+wiBBu/1Jynz6PiD9RUElAys7OJj8/n4KCApqamli2bBnz58/v0CYpKYm1a9cCcOzYMfbt20dKSood5QKw7fBx3tlbSu5lKQxVf34SQOw/Mixig+DgYBYvXszs2bPxer3k5OSQnp7OkiVLAFi0aBH/9m//xu233860adMwxvCb3/yGESNG2FbzH9/OJzbSyW2XJNtWg4gdrK721bdzxoki0pHb7SYvL6/Xl5t3sJJ/XrKBn82bRO5l43t9+SI26vFqde36ExkA/vj2fkYMCeU7s5LtLkWk3ymoRPzcxgMVfPRZBXd+dTzhTt1eXgKPgkrEz72+4whjh0dwy8XqeFYCk4JKxI/tPVrN85sOszA7kbAQbU1JYFJQifixp9cfIjQ4iIXZ2pqSwKWgEvFTVXXNvLatmOsyE4iJVJ9+ErgUVCJ+6qUthdQ3e7lV101JgFNQifghn8/wzIZDuMfGMHVMtN3liNhKQSXih9btL+VwZR236X5TIgoqEX/01PpDxA0NZc7U0XaXImI7BZWInzlQVsP7+8u45eKxhDj0ERXRp0DEzzyz4RAhDoubL07subFIAFBQifiRmkYPf99SxLxp8cQNDbO7HBG/oKAS8SOvbi3iZKNHJ1GItKOgEvETxhie3nCIaWOiuShxmN3liPgNBZWIn1j/eQWfldZw26XJWFaPt+gRCRgKKhE/8dT6g8RGOrkmI97uUkT8ioJKxA8UVtaxds8x9ZIu0gUFlYgfeGVrEQDfnjXW5kpE/I+CSsRmHq+PZzce4oYZLhKGhdtdjojfUVCJ2OzDz8opr2nia5Pj7C5FxC8pqERs9uq2YqLDQ7hikoJKpCsKKhEb1TR6eGvXUa7JiCc0WCdRiHRFQSVio9WfHqWh2ceCGWPsLkXEbymoRGz0ytYixg6PYEZSjN2liPgtBZWITUqq6tlwoIJvZI5RTxQiZ6CgErHJa9uOYAxcf5F2+4mciYJKxAbGGF7dVkTW2BiSR0TaXY6IX1NQidhg15Fq9h+r0daUyFlQUInY4NVtxTgdQeqAVuQsKKhE+pnH62P59iNcMWkkwyKcdpcj4vcUVCL97IPPyimvaeT6i1x2lyIyICioRPrZq1tPdZk00u5SRAYEBZVIP6pp9PCP3eoySeRcKKhE+tGqT0pau0zSbj+Rs6WgEulHr2wtJnl4BDOShtldisiAoaCSgLV69WrS0tJITU3lkUce6bLNunXryMzMJD09ncsvv/yCnu/IiXo2FlTwjYvUZZLIuQi2uwARO3i9Xu6++27WrFmDy+UiOzub+fPnM2XKlLY2J06c4K677mL16tUkJSVRWlp6Qc/52vZidZkkch60RSUBafPmzaSmppKSkoLT6WThwoUsX768Q5vnn3+eBQsWkJSUBEBc3Pnf2NAYw6tbi8kaG8PY4eoySeRcKKgkIBUXF5OYmNg27HK5KC4u7tBm//79HD9+nK9+9atkZWXxzDPPdLmspUuX4na7cbvdlJWVddlm15Fq8kvVZZLI+dCuPwlIxphO404/buTxeNiyZQtr166lvr6eSy65hFmzZjFx4sQO7XJzc8nNzQXA7XZ3+XyvbFWXSSLnS0ElAcnlclFYWNg2XFRUREJCQqc2I0aMIDIyksjISC677DJ27NjRKah64vH6eH1HMf80KU5dJomcB+36k4CUnZ1Nfn4+BQUFNDU1sWzZMubPn9+hzXXXXccHH3yAx+Ohrq6OTZs2MXny5HN+ro0HKqhp9HC9bjcvcl60RSUBKTg4mMWLFzN79my8Xi85OTmkp6ezZMkSABYtWsTkyZOZM2cOGRkZBAUFcccddzB16tRzfq5XthUTFRbCVyeqyySR82F1ta++nTNOFJGO3G43eXl5bcMNzV6yf/U2c6aO5nc3TrexMhG/1eNFhdr1J9KHPsgv52Sjh2umJ/TcWES6pKAS6UMrdh5hWEQIl44fbncpIgOWgkqkjzQ0e1mz+xhz0kcT4tBHTeR86dMj0kfW7SujtsnLNRna7SdyIRRUIn1kxSclxEY6mZUSa3cpIgOagkqkD9Q3eVm75xhzpo4mWLv9RC6IPkEifeDdfaXUNXm5Zpq6TBK5UAoqkT6wYmcJI4Y4uThFZ/uJXCgFlUgvq230sHbvMeZOjccRpBskilwoBZVIL3tnbykNzT6+rp7SRXqFgkqkl63YWULc0FCyk3W2n0hvUFCJ9CKfMby7r5R507TbT6S3qPd0kV5UXe/B49FuP5HepC0qkV5UVd/M6KgwspJi7C5FZNBQUIn0kuqGZk42NDNvWjxB2u0n0msUVCK95O3dxzCg3X4ivUxBJdJLVuwsIcQRxIykYXaXIjKoKKhEekFVXTPv55cRHR6CZWm3n0hvUlCJ9IJ/7D5Ks9cQHR5idykig45OTxfpBSs+KcEVE06D02F3KSKDjraoRC7Q8domPswv10kUIn1EQSVygf6x+ygen+GaabqTr0hfUFCJXKC8g8cZPzKSqWOi7C5FZFBSUIlcgIqaRl7ZVsyc9NE620+kjyioRC7Ayk+P4vUZrs3Ubj+RvqKgErkAb+w4woS4IaSNGmp3KSKDloJK5DyVVNXz8cFKrp2eoN1+In1IQSVynlbsLMEYuHa6dvuJ9CUFlch5emPHEaaNiWbciEi7SxEZ1BRUIufhYHktO4qquHa6LvIV6WsKKpHz8ObOIwBck6HdfiJ9TUElch5e33GE7OQYEoaF212KyKCnoBI5R/uOnmT/sRqdRCHSTxRUIufojR1HCLJg3jQdnxLpDwoqkXNgjOH1HUf4UuoIRgwJtbsckYCgoJKAtXr1atLS0khNTeWRRx7ptt3HH3+Mw+Hg5ZdfZmdRFYcr67hWJ1GI9BsFlQQkr9fL3XffzapVq9i9ezcvvPACu3fv7rLdAw88wOzZs4GW3X4hDovZU0f3d8kiAUtBJQFp8+bNpKamkpKSgtPpZOHChSxfvrxTuz//+c/ccMMNxMXF4TPw5s4SLp8Yp1vOi/QjBZUEpOLiYhITE9uGXS4XxcXFndq8+uqrLFq0CICCmiCOVjfoIl+RfhZsdwEidjDGdBp3eseyP/rRj/jNb36Dw+EAYHulg/AQB1dNGdWh3dKlS1m6dCkAZWVlfVSxSOBSUElAcrlcFBYWtg0XFRWRkNDxBIm8vDwWLlwIQHnlcYZHz2GGK4gIZ8ePTW5uLrm5uQC43e4+rlwk8CioJCBlZ2eTn59PQUEBY8aMYdmyZTz//PMd2hQUFLT9Pj/3AXaGD+XOeVn9XapIwFNQSUAKDg5m8eLFzJ49G6/XS05ODunp6SxZsgSg7bjUKaVhiYQ5DJenjbSjXJGAZnW1r76dM04UCQQNzV6yf/U2s6eO5vc3Tj9jW7fbTV5eXj9VJjIo9HjXUZ31J9KD9/aXcbLRw3z17SdiCwWVSA8+2F/G8Egnl44fbncpIgFJQSVyBlV1zby4pYgFM8YQ7NDHRcQO+uSJnMHrO4pp8vi4LnOM3aWIBCwFlcgZvJhXxOT4KNITouwuRSRgKahEurH3aDWfFFdxY5arU68VItJ/FFQi3Xgpr4gQh8U3LtJuPxE7KahEutDk8fHqtmKunDyK2Ein3eWIBDQFlUgX3tlbSmVtEze5E3tuLCJ9SkEl0oWX8gqJGxrKVyaMsLsUkYCnoBI5TWl1A+v2l3FDlkvXTon4AX0KRU7zyrZivD7DjVkuu0sRERRUIh0YY3gpr5CssTGkjBxidzkigoJKpIOth0/weVktN7m1NSXiLxRUIu28vKWQ8BAHX89QT+ki/kJBJdKqvsnLGztKmDctniGhuqeoiL9QUIm0WvVpCTWNHm7Ubj8Rv6KgEmn1Ul4RY4dHcPG4WLtLEZF2FFQiwOGKOjYcqOCfZ6gDWhF/o6ASAf6+pZBLUoazYIY6oBXxNwoqCXj1TV6e3niI6PAQxsRE2F2OiJxGQSUB79VtxZyoa+a7X0q2uxQR6YKCSgKaMYYnPyogPSGKmTqJQsQvKagkoH34WTn5pTV890vjdBKFiJ9SUElAe/Kjg4wY4uTa6fF2lyIi3VBQScA6UFbDO3tLueXisYQGO+wuR0S6oaCSgPX0+oM4HUHcMivJ7lJE5AwUVBKQquqbeWlLEddMjyduaJjd5YjIGSioJCC9lFdIXZOXnC+Ns7sUEemBgkoCjtdneGr9QWYmxzJ1TLTd5YhIDxRUEnDW7D5G0fF6XeArMkAoqCTgPLPhIGOGhXPVlFF2lyIiZ0FBJQHl44OV7DpSxQ+/lkqwQ29/kYFAn1QJKH9cs58Qh4P509VLushAoaCSgLHxQAXrP6/gzq+OJ9ypC3xFBgoFlQQEYwx/WLOfuKGh3HKxLvAVGUgUVBIQNnxeweaCSu766njCQlq2plavXk1aWhqpqak88sgjneZ57rnnyMjIICMjg0svvZQdO3b0d9kiAgTbXYBIXzPG8Me39zM6KoyFM1u2prxeL3fffTdr1qzB5XKRnZ3N/PnzmTJlStt848aN47333iMmJoZVq1aRm5vLpk2b7PozRAKWtqhk0Pvws3I+Pnicu6/4Ymtq8+bNpKamkpKSgtPpZOHChSxfvrzDfJdeeikxMTEAzJo1i6Kion6vXUQUVDLIGWP445r9JESHcVN2Ytv44uJiEhO/GHa5XBQXF3e7nMcff5y5c+d2OW3p0qW43W7cbjdlZWW9V7yIANr1J4Pce/vL2Hr4BL++fmqHW3kYYzq17e7Gie+++y6PP/44H374YZfTc3Nzyc3NBcDtdvdC1SLSnoJKBi2fz/DC5sOMGRbOjVmJHaa5XC4KCwvbhouKikhISOi0jJ07d3LHHXewatUqhg8f3uc1i0hn2vUng9ar24p5a9cxfjp3Es7gjm/17Oxs8vPzKSgooKmpiWXLljF//vwObQ4fPsyCBQt49tlnmThxYn+WLiLtaItKBqWTDc08snovmYnD+Pq0zreZDw4OZvHixcyePRuv10tOTg7p6eksWbIEgEWLFvHggw9SUVHBXXfd1TZPXl5ev/4dIgJWV/vq2znjRBF/9dDKPfz1gwO8dteXmJ44rN+e1+12K8xEzk3XB4fb0a4/GXQ+K63hiQ8LuCkrsV9DSkT6hoJKBhVjDL98YxfhTgf3zUmzuxwR6QUKKhlU1uw+xgf55fzrlRMZMSTU7nJEpBcoqGTQqG/y8p/rPmdC3BC+c8lYu8sRkV6is/5k0Hh41R72Ha3mv++YRYhuiigyaOjTLIPCun2lPLPhELdcPJassTF2lyMivUhBJQPe8dom7n95JxNHDeF/zdYJFCKDjXb9yYBmjOHnr33C8bomnrg9u613dBEZPLRFJQPaa9uLWfnJUf71qolMHRNtdzki0gcUVDJgFZ+o5/++tgv32Bi+f9l4u8sRkT6ioJIBqdnr47er9+Izhj/clIkjqMdeWERkgNIxKhlwjDH84vVdLN9+hP/6dhZJwyPsLklE+pC2qGTAeXr9QZ7bdJjvX57C7Kmj7S5HRPqYgkoGlHX7Snnwzd1cNWUUD8yeZHc5ItIPFFQyYOQfO8kPnt9G2ugo/vTNTIJ0XEokICioZEAorW7ge0/nERri4PHb3ESG6vCqSKDQp138XkVNI7c+sZmk2HB+cnUaCcPC7S5JRPqRtqjEr5WebGDh0o0crKhl0eWpXJSkfvxEAo22qMRvHatu4Oa/buRoVQNP3j6TS8YPt7skEbGBgkr80pET9XzrrxspO9nI0zkzyU6OtbskEbGJgkr8zmelJ/nxizuoqGni2TsuZoZ294kENB2jEr+yZvcxvvHYeiwLnvsXhZSIaItK/ITPZ3h0bT6Prs0nwxXNX27J0tl9IgIoqMQPHK9r4v6XdrJmzzFumOHi19dP1X2lRKSNgkps9c7eY/zvVz4hNW4Iv7h2CrddmoxlqccJEfmCgkpsUVXfzH+8uZuXtxSRNmooP50zmWku3fhQRDpTUEm/8nh9/E9eIW/uKGHzwUruuSKVH3wtldBg7eoTka4pqKRfGGP4x+5j/Gb1Xg6U1ZKdHMOKH3yZSfFRdpcmIn5OQSV9yhjDpgMV/P4f+8k7dJzxIyNZ+p0srpoySseiROSsKKikT9Q3eVm+vZhnNhyivKaBEIeDh66fxk1uF8EOXb4nImdPQSW96mB5Lf+98RAv5hVS3eBh0uih/OjKNK6/aAzhTh2HEpFzp6CSC1ZSVc9bnx5l1adH2Xv0JLWNHuZMHc2tlySTnRyjXXwickEUVHLOvD7DnpJqPj5YyfLtR9heeAKACXFD+PFVE5kzdTSjosLsLVJEBg0FlfTI4/Wx7+hJNhZUsvFABZsLKqmqb2bmuFg8Ph/3zU5jdvpoUuOG2F2qiAxCCirpoL7Jy/5jJ9ldUs2nxVXsOlLN3qPVJA+PZO/Rk4wdHsHcqaOZlTKcWSmxjI5Wf3wi0rcUVAHG5zOU1zZytKqBkqoGCivrOFBey8HyWgrKaympaiApNoLDlXUMDQtmSnwUt1w8lpnJMUxzDRtUHcWuXr2ae++9F6/Xyx133MFPf/rTDtONMdx7772sXLmSiIgInnrqKWbMmGFTtSKBS0E1wDV7fZxs8FBd38zJBg81jc2U1TRRWdNIZW0TFbVNVNY24QwOYsuh4xyrbqDZawAIcVg0ew3DIkIYNyKSS8YPZ9zwSCYnRDExbiiJseGD9kQIr9fL3XffzZo1a3C5XGRnZzN//nymTJnS1mbVqlXk5+eTn5/Ppk2buPPOO9m0aZONVYsEpkEdVMYYjGn9/dQwYHwGY/HFNAMGg8/XOh0wPvAZg691Hl9LI3wGfMaHt216y8kFPtMyv6f1d4/PYIyh2WvweA0en6/1p8FnfDR6fDR7DE1eH81eHw7Loqq+mUaPj0aPl4bmlp8hjiDKTjZS3+ylttFDXZO37REVHsyBstoOf/PM5Fg2H6wEwLIgJsJJbKSTrKRhuMfGMDo6nIRhYYyOCiM+OhxXTDgxkc5+ekX8x+bNm0lNTSUlJQWAhQsXsnz58g5BtXz5cm699VYsy2LWrFmcOHGCkpIS4uPj7SpbJCD1S1DtO3qS+Ys/7HKaKyacwuP1HcbNSBzG1sMngJYAAch0DWNb4alx7eYfFk7h8bq2cabdxPEjI/n8tC9ygIuShrGtdfntDY90UlHb1GHctDHRfFJc1WFc1tgYthw63mn+6YnR7Cj8ou2IIU7Ka5o6tUuIDuNIVUOHcenxUewqqSbIgrAQB6HBQYQGOxgTE0Zto5cIp4MIZzDDh4QS6XQQ7gwmbmgojiCLqLBgosJDGBoWwrCIEIaFhxAb6WRYhBNH0ODcIrpQxcXFJCYmtg27XK5OW0tdtSkuLlZQifQzy7T/Zj/NnDlzTHl5+QU/icdrOFx2nIjwiM4FWB3DBcARZOE9baTD6jju1NevZVl09zcEWRY+Y7BOtbZOjW/ZMmq/nFO1nK6utpbIIUM6tA2yvgjLU8u2rNb5zanlWG3Ls1rrbB2N1VpbyzxW6/R2bXpJWVkZI0eO7MUl9p++rv348eNUV1czduxYACoqKqitrSUpKamtTX5+PvHx8Qxpff3379+Py+UiIqLj+7isrIxTn5PGxkYyMzP7rO6+NFDfLwO1bhi4tfdm3Vu2bHnLGDPnjI1ado91++g1WVlZvbm4fjNQ6zZGtZ/J+vXrzdVXX902/NBDD5mHHnqoQ5vc3Fzz/PPPtw1PnDjRHDly5IzLjYiI6N1C+9FAfb8M1LqNGbi193LdPeUQ6nRNAlJ2djb5+fkUFBTQ1NTEsmXLmD9/foc28+fP55lnnsEYw8aNG4mOjtZuPxEbDOqTKUS6ExwczOLFi5k9ezZer5ecnBzS09NZsmQJAIsWLWLevHmsXLmS1NRUIiIiePLJJ22uWiQw9VtQ5ebm9tdT9aqBWjeo9p7MmzePefPmdRi3aNGitt8ty+Kxxx47p2WOGDGiV2qzw0B9vwzUumHg1t7fdZ/xZAo6nmAnIj1wu93k5eXZXYbIQNLjOWQ6RiUiIn6tV4PqpZdeIj09naCgoE7/VT788MOkpqaSlpbGW2+91eX8lZWVXHXVVUyYMIGrrrqK48c7X6vU1775zW+SmZlJZmYmycnJ3Z5qnJyczLRp08jMzMTtdvdvkd34xS9+wZgxY9rqX7lyZZftVq9eTVpaGqmpqTzyyCP9XGVn9913H5MmTSIjI4Prr7+eEydOdNnOn9Z5T+vQGMMPf/hDUlNTycjIYOvWrTZU2VFhYSFXXHEFkydPJj09nUcffbRTm3Xr1hEdHd32HnrwwQdtqLRrPb3+/rjOAfbt29e2PjMzM4mKiuJPf/pThzb+st5zcnKIi4tj6tSpbePO9nu5T79Xejgt8Jzs3r3b7N2711x++eXm448/bhu/a9cuk5GRYRoaGsyBAwdMSkqK8Xg8nea/7777zMMPP2yMMebhhx82999//7mW0Kt+/OMfm1/+8pddThs7dqwpKyvr54rO7N///d/N7373uzO28Xg8JiUlxXz++eemsbHRZGRkmF27dvVThV176623THNzszHGmPvvv7/b191f1vmZ1uGp03ZXrFhh5syZY3w+n9mwYYOZOXOmnSUbY4w5cuSI2bJlizHGmOrqajNhwoROr/27775rvv71r9tRXo96ev39cZ2fzuPxmFGjRpmDBw92GO8v6/29994zW7ZsMenp6W3jzuZ7+QK/V/r39PTJkyeTlpbWafzy5ctZuHAhoaGhjBs3jtTUVDZv3txlu9tuuw2A2267jddee603yzsnxhhefPFFbr75Zttq6Avtuw5yOp1tXQfZ6eqrryY4uOW8nlmzZlFUVGRrPT05m3XYXfdLdoqPj2/rVHfo0KFMnjyZ4uJiW2vqTf64zk+3du1axo8f33ahub+57LLLiI2N7TDubL6X+/p7pV+OUXXXFc3pjh071nadSnx8PKWlpf1RXpc++OADRo0axYQJE7qcblkWV199NVlZWSxdurSfq+ve4sWLycjIICcnp8tN9LN9LezyxBNPMHfu3C6n+cs6P5t16O/r+eDBg2zbto2LL76407QNGzYwffp05s6dy65du2yorms9vf7+vs4Bli1b1u0/v/663s/me7nP1/3ZbHa1fwBvA5928biuXZt1gLvd8GPAt9sNPw7c0MWyT5w2fPxc6+vFv+EvwE/OsIyE1p9xwA7gsr6o9VxqB0YBDlr+Afk18EQX898I/K3d8HeAP9tZd7s2PwdepfVsVH9Z5+eyDoHVrT9XAF9u12YtkGVHvV3UPwTYAizoYloUMKT193lAvt31nu3r78/rvLUeJ1AOjPLn9Q4kA5+2Gz5x2vTjXczTp98r53wdlTHmynOdBygCEtsNu4AjXbQ7ZllWvDGmxLKseKBPNql6+hssywoGFgBZZ1jGkdafpZZlvQrMBN7vzTq7ed6zWv+WZf0VeLOLSWf7WvSqs1jntwHXAF8zre/0LpZhyzrvQrfr0HzRZ5kt67knlmWFAH8HnjPGvHL6dGNMdbvfV1qW9Z+WZY0wxlx4p58X6Cxef79c5+3MBbYaY46dPsGf1ztn973cp+u+v05Pfx1YaFlWqGVZ44AJQOeDVC3tbmv9/TbAroMnVwJ7jTFdHiyxLCvSsqyhp34HrqZl68BWrW+iU66n65o+BiZYljXOsiwnsJCW9W4by7LmAA8A840xdd208ad1fjbr8HXgVqvFLKDKGGPrAROr5eZijwN7jDF/6KbN6NZ2WJY1k5bviIr+q7JrZ/n6+906P83NwAtdTfDX9d7qbL6X+/Z7pZc3Ga+nJVkbgWPAW+2m/Rz4HNgHzG03/m+07iYEhtOyuZ7f+jPWpk3fp4BFp41LAFa2/p5Cy66HHcAu4Od21NlF3c8CnwA7W98k8afX3jo8D9jf+nrYXjvwGVAIbG99LPH3dd7VOgQWnXrf0HIR42Ot0z+h3a5wG2v+Mi0X8e9st67nnVb3Pa3rdwewEbjU7rrP9Pr7+zpvV38ELcET3W6c3613WoK0BGhu/S7/Xnffy/35vdJTzxQiIiK2Us8UIiLi1xRUIiLi1xRUIiLi1xRUIiLi1xRUIiLi1xRUIiLi1xRUIiLi1xRUIiLS7yzLWmRZ1vbWR4FlWe9221YX/IqIiF1a+598B/itMeaNrtpoi0pEROz0KPBOdyEFnHvv6SIiIr3BsqzbgbG09HXYfTvt+hMRkf5mWVYW8DTwFWNM57u8tqNdfyIiYod7gFjg3dYTKv7WXUNtUYmIiF/TFpWIiPg1BZWIiPg1BZWIiPg1BZWIiPg1BZWIiPg1BZWIiPg1BZWIiPg1BZWIiPi1/w8HOE2NJtMbnAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sympy.plotting.plot(logistic);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a well-groomed $S$-shaped function: it's called a _sigmoid_ curve. Notice that when $z=0$ it takes the value $0.5$, and it approaches zero on the left, and one on the right.\n",
    "\n",
    "Let's generate some synthetic data to play with. (We take this example from the SciPy 2019 tutorial by Eric Ma [1]). \n",
    "Our goal is to use gradient descent to find the model parameters $w$ and $b$ that best fit the data, in some sense that we need to discover."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcxklEQVR4nO3de2xc55nf8e/DmSGH5PAiXixZlmRKip1YdpM2mZXdxN5Nk2btXBC32C0cp3WbtIBhIA6yxQZN2kXbBfafFt20m8Um8ApBsF13sXaATbNJ4K2bAN22yla7phLnYvkmyZZFS5RISbzPkDPi0z+GMzocDYeH5Fw4h78PEGjmzDtnXlrKj8+8533fY+6OiIi0vrZmd0BERGpDgS4iEhEKdBGRiFCgi4hEhAJdRCQi4s364KGhIR8ZGWnWx4uItKSTJ09OuvtwpdeaFugjIyOMjo426+NFRFqSmZ1b6zUNuYiIRIQCXUQkIhToIiIRoUAXEYkIBbqISESsO8vFzL4JfAK47O73VHjdgK8CHwMWgM+4+49r3VERaY43JuY4fuYKl6az7O5Lcv/hQQ4Op+pyzvLjI7s6efNapmq7WBsYkF+maY/L+xr2PbX4bxlk6+22aGa/DMwBf7xGoH8M+DyFQL8X+Kq737veB6fTade0RZHto1JITswu8da1Be7a08OBwW7msnmmMjkeSe8D2FSolp/zrcl5To3PMtAV51omv+r435y7xtGRXWu262qPceLsVRy4a0+KV8bnGv74/YcHmM9eL/U1TJ/ef3iAjni89N9yI6FuZifdPV3xtTDb55rZCPD9NQL9D4G/dPc/XXn+KvBBd79Y7ZwKdJHGqFZhF1975cJMKWSDgdTd0cbysrHskB7pB4dfXJjh2vwSyfbYpkI1eM7Dw52cmVigzYzJ+UWGujtKn3X68jwzmRy9XQneMdTN6LlrN7WLxyARi2HA29ML7O3ravjj9kQbOKW+4rCYX173PfcdHGQmk6M7Geex+24P/fdZLdBrMYZ+G3A+8Hxs5ViljjxuZqNmNjoxMVGDjxbZ2d6YmOPpE+f43edf5ekT53hjYu6m158dHWM+m2dPX5L5bJ5nR8d4Y2Ju1Wsz2RwxM14Zn+Nnb0/T19XOrq52zl/N0teZoKs9xovnpxg9dw0MrmWWKrY/eX5q3cfBc46+NUVne5y+rgQzmVzp+OmJeWayOXo748xkcpyemK/Y7vzVDMl4Gx2JNqYXck15PJPJrerrTDbcewBSyTiXprM1+/dQi0C3Cscqlv3ufszd0+6eHh6uuHJVREKqFtZFx89cob8zQW9ngjYzlvLXOTsxx29/7xRf+cFrLC8v09uZYHYxXzEkAbL5ZToSbZy/mqGzPU4bhmGbDtXgOYvHF3PL9HUlSsdnMjl6kwlmMnl6OxOlkCxvVzxX8HijH/d2Jlb1tTcZ7j0Ac9k8u/uSNfs3UYtAHwP2B57vAy7U4LwiUkW1sC5W65ems6SShbkPk7PZUoUNztW5JV6+OMvkXLYUQuUhuX+gk8xSnumFXKFKc2dh6Tr7Bzo3HarBc/Z2JpjO5FhYuk76QH/peE8yzp6eDqYyOfb0dNDTEa/Ybt+uJNMLS1xbWOJ9+/ub8vjwUNeqvh4a6gr1nplMjqlMjvsPD9bs30QtxtA/DjzJjYuiv+/uR9c7p8bQRbbmd59/lT19ycK48kpYJ9tjLOau83cPDTGVyZGMG8lEnN7OBCfOXimN7ZaP+wbHqM2czNLyqgt+p8ZnWcpfp7+7nXv29oJTsf1GLiKeGp9lsCvO1bILoafGZzk42MWdt/aWZo4Ex/jL2w2k2nfULJdqY+hhpi3+KfBBYMjMxoB/DyQA3P0p4DkKYX6awrTFz26odyKyKbv7ksxlC1/zi2PMBvR1tZe+0mdyhVkpUAjvjngbC7lljuztAYcXzl1jYjbL0ZEB3rW756aQzF2H/UPdPHrvAQCeHR2jPRYjlYxXbJ9fhnsPDVR9HDxncAri+HR21fGiB1b+XK/ddvDA+k3qKlSFXg+q0EW2pjiG3r9SfXfE28jklkmP9DOUSrLszvh0ll97720cP3OFH54apyMR4569vQylCuO2b07McWEmy8hgd6iKsR5z0mVjtlShi8j2dHA4xSPpfRw/cwXHcaMU5nDjgtvB4RQHh1Pcf3iwVGEvuzOXzdMWa+M3P3Jn6FAunku2JwW6yDYVphquFtZTmRwP3r17VdviL4DxlXM+ePduBXSEaMhFpEE2MlwRHE5JJeOrVmhWe4+GQ6JPQy4iTRYM6D0rFzOfHR1bM6CDUxKB0p/Hz1xZM6Q1HCIKdJEG2GhAX5rOsiew4GRyNsvrl+e4MJ0B0MVLqUjb54o0QHCBT1Fx2Xel5fvFKYlwY0HQbDbP3v7OiitCg8KsIJVoUoUuUmOVquPgnPGiuWyeWBsVh2I+cGiAH529CsDrl+dos8JmVnfcklq3ut/McI1Egyp0kRpaqzoe2dXJVCbHTCbHsntp2bdBxeX73/jRm3TEjWwuz4XpDD2d8VVTEqtt6lTt24BEmwJdpIbW2l8lGNDj01m6k3EeSe8jv8yae610JuJk884D7xji7lv7SmEO1Td1Cg7XhGkv0aFAF6mhapthFQP61957G4/ddzsHh1Orwre4fL8NKy3f7+9M4FCxul9rU6f7Dw9uqL1EhwJdpIbCBPTxM1dK7YPhO5PJlXYzfMdwN1Co3q8vwyPpfXQn46uq+2rTFzfSXqJDF0VFaqi4YhMqbIZFIaDHA2PZG12+H5bmpO9MCnSRGgob0OXvCbt8X6QaBbpIjYUJ6LUW/mivFdkKBbpInawV0FB57nlxnFsBLpulQBepo0oB/fSJc1ta+KNl/bIWzXIRabCtLPzRsn6pRhW6yBZttGJeaxuAMAt/tKxfqlGFLrIFm6mYt7LwR8v6pRoFusgWlC/1r7R4qNxWFv5oWb9UoyEXkS0o37ccbl48VMlmZ7MEFy4F72SkueoCqtBFtqTRFbOW9Us1qtBFqljvgmczKmbNVZe1KNClpdVzTnaY+4BqdadsJwp0aVkbvfHyRoWdIqiKWbYLjaFLy9rMDJON0BRBaTUKdGlZ9Q5cTRGUVqNAl5ZV78DVnX+k1SjQpWXVO3A1RVBajS6KSstqxAwTXfCUVqJAl5amwBW5IdSQi5k9ZGavmtlpM/tyhdf7zOx7ZvZTM3vJzD5b+66KiEg16wa6mcWArwEfBY4Aj5rZkbJmnwNOuft7gA8CXzGz9hr3VUREqghToR8FTrv7WXdfAp4BHi5r40CPmRmQAq4CeUREpGHCBPptwPnA87GVY0F/ANwFXAB+DnzB3ZfLT2Rmj5vZqJmNTkxMbLLLIiJSSZiLolbhmJc9fxB4EfgQcBj4gZn9X3efWfUm92PAMYB0Ol1+DpG60/04JcrCVOhjwP7A830UKvGgzwLf9oLTwBvAu2rTRZHa0P04JerCBPoLwB1mdnDlQuengO+WtXkL+DCAme0G3gmcrWVHRbaq3nu/iDTbukMu7p43syeB54EY8E13f8nMnlh5/Sngd4A/MrOfUxii+ZK7T9ax3yIbttm7C4m0ilALi9z9OeC5smNPBR5fAH61tl0Tqa3i3i/FbXBBm21JtGgvF9kxtNmWRJ0CXXYMbbYlUae9XGRH0d4vEmWq0EVEIkKBLiISERpyEQlJq0xlu1OFLhKCVplKK1Cgi4SgVabSChToIiFcms6SSq4eoUwl41zSKlPZRhToIiEUV5kGaZWpbDcKdJEQtMpUWoECXSQErTKVVqBpiyIhaZWpbHeq0EVEIkKBLiISEQp0EZGIUKCLiESELoqKoH1aJBpUocuOp31aJCoU6LLjaZ8WiQoFuux42qdFokKBLjue9mmRqFCgy46nfVokKhTosuNpnxaJCk1bFEH7tEg0KNBFNkHz1mU70pCLyAZp3rpsVwp0kQ3SvHXZrhToIhukeeuyXSnQRTZI89ZluwoV6Gb2kJm9amanzezLa7T5oJm9aGYvmdn/rm03RbYPzVuX7WrdQDezGPA14KPAEeBRMztS1qYf+DrwSXe/G/hHte+qyPageeuyXYWZtngUOO3uZwHM7BngYeBUoM2ngW+7+1sA7n651h0V2U40b122ozBDLrcB5wPPx1aOBd0J7DKzvzSzk2b2TyudyMweN7NRMxudmJjYXI9FRKSiMIFuFY552fM48D7g48CDwL81sztvepP7MXdPu3t6eHh4w50VEZG1hRlyGQP2B57vAy5UaDPp7vPAvJn9H+A9wGs16aWIiKwrTIX+AnCHmR00s3bgU8B3y9r8OfCAmcXNrAu4F3i5tl0VEZFq1q3Q3T1vZk8CzwMx4Jvu/pKZPbHy+lPu/rKZ/Q/gZ8Ay8A13/0U9Oy4iIquZe/lweGOk02kfHR1tymeLiLQqMzvp7ulKr2mlqIhIRGj7XJEy2hpXWpUqdJEAbY0rrUyBLhKgrXGllSnQRQK0Na60MgW6SIC2xpVWpkAXCdDWuNLKFOgiAdoaV1qZpi2KlNHWuNKqVKGLiESEAl1EJCIU6CIiEaFAFxGJCAW6iEhEKNBFRCJCgS4iEhEKdBGRiFCgi4hEhAJdRCQiFOgiIhGhQBcRiQgFuohIRCjQRUQiQoEuIhIRCnQRkYhQoIuIRIQCXUQkInQLOmm4NybmOH7mCpems+zuS3L/4UHd8k2kBhTo0lBvTMzx7OgY/Z0J9vQlmcvmeXZ0rOY3YtYvDdmJNOQiDXX8zBX6OxP0diZoM6O3M0F/Z4LjZ67U7DOKvzTms3n29CWZX/ml8cbEXM0+Q2Q7UqBLQ12azpJKrv5imErGuTSdrdlnNOKXhsh2FCrQzewhM3vVzE6b2ZertPslM7tuZr9euy5KlOxeGWYJmsvm2d2XrNlnNOKXhsh2tG6gm1kM+BrwUeAI8KiZHVmj3X8Enq91JyU67j88yFQmx0wmx7I7M5kcU5kc9x8erNlnNOKXhsh2FKZCPwqcdvez7r4EPAM8XKHd54E/Ay7XsH8SMQeHUzyS3kd3Ms74dJbuZLzmF0Qb8UtDZDsKM8vlNuB84PkYcG+wgZndBvxD4EPAL611IjN7HHgc4MCBAxvtq0TEweFUXWecFH9pHD9zhfGVWS4P3r1bs1wk8sIEulU45mXPfw/4krtfN6vUfOVN7seAYwDpdLr8HBJhjZ5GWO9fGiLbUZghlzFgf+D5PuBCWZs08IyZvQn8OvB1M/sHteigtD5NIxRpjDAV+gvAHWZ2EHgb+BTw6WADdz9YfGxmfwR8392/U7tuSisLTiMESn8eP3NFVbRIDa1bobt7HniSwuyVl4FvuftLZvaEmT1R7w5K69M0QpHGCLX0392fA54rO/bUGm0/s/VuSZQUpxEWK3PQNEKRetBKUak7TSMUaQxtziV116hphNqQS3Y6Bbo0RL2nETZqF0eR7UxDLhIJ2pBLRIEuEaGZNCIKdIkIbcglokCXiNBMGhEFukREI3ZxFNnuNMtFIkMbcslOpwpdRCQiFOgiIhGhQBcRiQgFuohIROiiqOxY2vtFokYVuuxIuouSRJEqdIm8SpW47qIkUaQKXSJtrUr81Ysz2vtFIkeBLpG21i6MU5mc9n6RyFGgS6SttQtjXzKhvV8kchToEmlr7cL4rr292vtFIkcXRSXS7j88yLOjY0ChMp/L5pnK5Eq3wFOAS5SoQpdI0y6MspOoQpfIUyUuO4UqdBGRiFCgi4hEhAJdRCQiFOgiIhGhQBcRiQgFuohIRCjQRUQiIlSgm9lDZvaqmZ02sy9XeP0fm9nPVv73V2b2ntp3VUREqlk30M0sBnwN+ChwBHjUzI6UNXsD+BV3fzfwO8CxWndURESqC7NS9Chw2t3PApjZM8DDwKliA3f/q0D7E8C+WnZSdgbdEk5ka8IMudwGnA88H1s5tpZ/AfxFpRfM7HEzGzWz0YmJifC9lMjTLeFEti5MoFuFY16xodnfoxDoX6r0ursfc/e0u6eHh4fD91Iib60bURw/c6XZXRNpGWECfQzYH3i+D7hQ3sjM3g18A3jY3fX/QtmQtW5EoVvCiYQXJtBfAO4ws4Nm1g58CvhusIGZHQC+DTzm7q/VvpsSdWvdiEK3hBMJb91Ad/c88CTwPPAy8C13f8nMnjCzJ1aa/TtgEPi6mb1oZqN167FE0v2HB3VLOJEtMveKw+F1l06nfXRUuS83aJaLyPrM7KS7pyu9phtcyE2aFay6EYXI1ijQZZXi9MH+zgR7Vsa1nx0dq9tt21SVi9SOAl1WCU4fBEp/Hj9zZcNBu15YN/qXh0jUaXMuAQrh+vSJc3znJ2O8dHGaybkb0wU3M30wzEIhzT0XqS0FuqwK3719ncxm8oy+OVUK9c1MHwwT1pp7LlJbGnKJoI2OSwfD945bUoyeu0abGa9fnqM9FmMqk+PBu3dvqA+XprPsKfslkErGGQ+EdXHueXFYBzT3XGQrVKFHzGb2RAlWykM9SdK376InGefiVJbuZHxTY9phFgpp7rlIbalCj5jNXNQsr5SHepK0x2McPTTIY/fdvql+3H94kGdHx4BCZT6Xzd9U6R8cTvFIeh/Hz1xhfOXbxIN37+bgcEqzX0Q2QYEeMWGGOsqFCd+NqhbW5e3Kj2n2i8jmKNAjZjPj0mHDd6M2u1CollMnRXYSBXrEbLba3k6rNMu/ZUzOZnn98hwXpjMAGn4RWYMuikZMsdruXhlm2exFzWYKXlCdnM0yeu4as9k8e/s7deMLkSpUoUfQdqq2NyP4LeP1y3O0mbHscMctKXo7E1ydW+QrP3iNkcFuXTAVCVCFLttO8FvGhekMPZ1x0iP9DKWSTM5mefnSLFfmF3WrOpEyqtAltEZOJQx+y5gPXOQ9PTFPzIxdPcnSClTQBVMRUIUuITXrJs7li48mZxe5vuy8Y7i71EbbBYgUqEKXUMJOJax1FV8+pXIg1c7eviRDqRuzYLRdgEiBAl1CCbNgqV4LgoLDL8XPmMnkarYISiQqNOQioYTZm6UR2+FGYVqmSL2oQpeqikMor1yY4a1rC9y1p4cDg90VK+PNbDuwGa0+LVOkXhTosqbgEMpde3vpbo9x6uIsmaXr3Hlr703bA2g7XJHmUqDLmsovhI4MpxhIddCdjFfchbEem3yJSHgaQ5c1bfSOQhrfFmkuVeiyps3u3KgAF2kOVeiyJt1RSKS1qEKXNemOQiKtRYHewjYaqsH2sTYwIL9M1ffqjkIirUNDLi1qo3urBNvH2+Cvz17l/529SiLGhvdlacQCIhHZOFXoTbCV4Yrie394apyORIx79vbSZol1dx0MhvCpizP0dbVjwJnJBe47OFj1veV0RyGR7UkVeoNtZdfC4HsNwxxG35xicq4wjbDalMLgFMSZbI5kvI2ORBszmdy67y2nOwqJbE+q0DchzFh0eRU+squTN69lbqqsl/LXOTsxx29/7xR/e39f6VyVzhussns7Eyzml+lqj3F6Yh4cfnFhhsX8Mk+fOHdTP37+9hSvXS58bm8yQTa/jHFj18SNrOhc745CoP3JRZrB3H39RmYPAV8FYsA33P0/lL1uK69/DFgAPuPuP652znQ67aOjoxvqbLUgLQZm+Wu1fjwxu1Ta06SrPcaJs1dx4P2HB5jPXufU+CwDXXGuZfKlfU/empznb85d4+jILl67NEdHvI1MbpnDw52cmVgg2R7j2twisbY2HLhrT4pXxuduOu/UwhIHh7u545YUOIyeu3bTe99/eICOeJypTI4PHBrgR2ev0t+ZIJvLl/pafv5i+41c1Cz+XXznJ2Ps7e/kjltSpS1tl90Zn87yxQffuaG/XxFZn5mddPd0xdfWC3QziwGvAR8BxoAXgEfd/VSgzceAz1MI9HuBr7r7vdXOu9FAD86sCIZTMfCKgRkM2WBw1epxd0cby8uFijQeg0QshgFL16+Tv+60mTE5v8hQdwfLDumRfk5fnmcmk6O3KwEOiyvV8dvTC+zt66r6uNp5i1X52cl5Dg13c8/e3lKozmRy/OLCNPfs7StVzZOz2VIVH/w2sJVph0+fOLfqjkLFz15rewAR2ZpqgR5myOUocNrdz66c7BngYeBUoM3DwB974bfDCTPrN7Nb3f3iFvteUu2iHg79nQnGZxfBKb128vxUKRhr9fjMxBxHbu1lMb9ceowVjh8aTpFMtHF2co7DQykW88ucnphnJpujtzPOTCbHe/f3lyrr6YUce3udhdwyXYkYyXjha8D0Qo5Dg203nXdhKc+yF8L99ctz3H1rH4eGU9zS08G7bu2lzaz03yuVjHNxOsN9h24sAhrqSfLLd3YwPp3lX36kNtWz9m8R2T7CXBS9DTgfeD62cmyjbTCzx81s1MxGJyYmNtTRahf1goEZfG16ofaPAbL55VWPF3PLACTjbSzmlunrSpTazGRy9CYTzGQKVexQT5L07bvAoT0ew61Qbe/u7Sydq/j+8vPu6e8kffsuepJxLk7d2Cvlnbf2Vtyr/Na+znX3MN8q7d8isn2EqdCtwrHycZowbXD3Y8AxKAy5hPjskuC+Ijdd1HNWDWkUXysGYy0f7x/oJLOUZzFn7NuVZHphCQf2DySZzuQKwyEH+jkzscBizujpjLOnp4O3ri5w5+4Uy+60x2McGk7x2L0H+NHZq7THYhwa6ioNFb1vf39piCd43iN7exhKJWmPxzh6aHDVkEalKvmR9+3jR2ev3nS81tWz9m8R2R7CBPoYsD/wfB9wYRNttiT41T4YfsUx9GJgBsfQg8FYq8fBi5QHB7sYSLXfdLH0wGA3ibYYp8Zn6U8m2D/UzQN3DPHmtcxNS+j3DXRx/MwV5hbz3HtooDSuXXwcPO9Ad0dpP5VgKFdbol88f/lxEYmeMBdF4xQuin4YeJvCRdFPu/tLgTYfB57kxkXR33f3o9XO26qzXKpdRKzX/ibaN0VEirZ0UdTd82b2JPA8hWmL33T3l8zsiZXXnwKeoxDmpylMW/xsrToftN5X+wfq8aEbUK+hBw1piEgYoRYWuftzFEI7eOypwGMHPlfbromIyEZo6b+ISEQo0EVEIkKBLiISEQp0EZGICLU5V10+2GwCONeUD9+aIWCy2Z1oMP3M0bfTfl5o3Z/5dncfrvRC0wK9VZnZ6FpzQKNKP3P07bSfF6L5M2vIRUQkIhToIiIRoUDfuGPN7kAT6GeOvp3280IEf2aNoYuIRIQqdBGRiFCgi4hEhAJ9C8zsi2bmZjbU7L7Uk5n9JzN7xcx+Zmb/3cz6m92nejGzh8zsVTM7bWZfbnZ/6s3M9pvZ/zKzl83sJTP7QrP71ChmFjOzn5jZ95vdl1pRoG+Sme2ncOPst5rdlwb4AXCPu7+bwt74/7rJ/amLlRuifw34KHAEeNTMjjS3V3WXB37T3e8C7gM+twN+5qIvAC83uxO1pEDfvP8C/Csq3Govatz9f7p78eakJyjckSqKSjdEd/cloHhD9Mhy94vu/uOVx7MUAu6m+wFHjZntAz4OfKPZfaklBfommNkngbfd/afN7ksT/HPgL5rdiToJdbPzqDKzEeDvAH/d5K40wu9RKMiWm9yPmgp1g4udyMx+COyp8NJvAf8G+NXG9qi+qv287v7nK21+i8JX9D9pZN8aKNTNzqPIzFLAnwG/4e4zze5PPZnZJ4DL7n7SzD7Y5O7UlAJ9De7+9ysdN7O/BRwEfmpmUBh++LGZHXX38QZ2sabW+nmLzOyfAZ8APuzRXbxQ95udb0dmlqAQ5n/i7t9udn8a4APAJ83sY0AS6DWz/+bu/6TJ/doyLSzaIjN7E0i7eyvu2haKmT0E/GfgV9x9otn9qZcwN0SPGitUJf8VuOruv9Hk7jTcSoX+RXf/RJO7UhMaQ5cw/gDoAX5gZi+a2VPrvaEVrVz4Ld4Q/WXgW1EO8xUfAB4DPrTyd/viSuUqLUgVuohIRKhCFxGJCAW6iEhEKNBFRCJCgS4iEhEKdBGRiFCgi4hEhAJdRCQi/j84Y6znxREyDgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# synthetic data\n",
    "x_data = numpy.linspace(-5, 5, 100)\n",
    "w = 2\n",
    "b = 1\n",
    "numpy.random.seed(0)\n",
    "z_data = w * x_data + b + numpy.random.normal(size=len(x_data))\n",
    "y_data = 1 / (1+ numpy.exp(-z_data))\n",
    "\n",
    "pyplot.scatter(x_data, y_data, alpha=0.4);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, we chose the parameters $w$ and $b$ for the model, and used them to get the intermediate variable $z$ adding some random noise to make our synthetic data look more \"realistic.\"\n",
    "The call to `numpy.random.seed()` makes our added noise reproducible, i.e., you always get the same pseudo-random numbers from the subsequent call to `numpy.random.normal()`. \n",
    "(Let's not get side-tracked into a discussion about pseudo-random number generation, and leave that for another tutorial.)\n",
    "\n",
    "We can apply a decision boundary now to assign the data to the two classes. Be sure to read the documentation of [`numpy.where()`](https://numpy.org/doc/stable/reference/generated/numpy.where.html) to understand the code below, noting that after the logical condition we specify the values to assign if the condition is `True` or `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQ+ElEQVR4nO3df2zc9X3H8ec7thPbDXEg8ZJCsiZEdCNi7Sa8wApb2bq24YeKJk0qdGMrm4SQSNVKq1a6aj+katqmaVupoEojhLp1VemkspVVdKyT1k1tlRWH0R8pTZWGAiY0GEjshdiNTd774y7h6px9Z/vO53zyfEhI3x+f+97rxJeXPnzve/5GZiJJOvet6HQASVJrWOiSVAgLXZIKYaFLUiEsdEkqRHen3nj9+vW5ZcuWTr29JJ2T9u3b92JmDtbb17FC37JlC8PDw516e0k6J0XE07Pt85KLJBXCQpekQljoklQIC12SCmGhS1IhGt7lEhEPADcBL2TmFXX2B3APcANwAnhvZj7e6qBSPU+NHuerP3iJI2OTbBjo5dpt69g6uHre42u3d62AAKZPMe/lDQO9bLmwjx8enVj0sdq9vNyzLvd8i83azPk6X9Hory1GxK8Ax4F/mKXQbwDeR6XQrwLuycyrGr3x0NBQetuiFuOp0eN8bniEtX09rO7t5vjkNMcmpnj30Ka6/5HMNv6aSy/ia4deZm1fD5NT0+w99DIJXL5xNd/70fGml9+y7SJemXyVbzx9lB1bLqR/ZdeCj9Xu5eWedbnnW2zWt2y7iFXd3XOer7OJiH2ZOVR3XzN/PjcitgBfnKXQPwl8JTM/W10/AFyXmc/PdUwLXYv16b1P88rkNGv6es5sG5+Y4nW93dx29RuaHv+dw2NccfEAa/p62HvoJX48fYoAnhs7wcUD/U0vr+xZAVk55pr+HkgWfKx2Ly/3rMs932KzruxZwdVb1815vs5mrkJvxTX0S4Bna9ZHqtvqBbkjIoYjYnh0dLQFb63z2ZGxSVb3/uRVw9W93RwZm5zX+OfHJs5sH5+cord7Bat6VjB2Yn7L4xNTjE9Osaav+8zyQo/V7uXlnnW551ts1vGJqYbn60K04peiUWdb3Wl/Zu4B9kBlht6C99Z5bMNAL8dnzLiPT06zYaB3XuNfP9B3Zvua3h4mq7Orgf75La/p6zlrprbQY7V7eblnXe75Fpv19Dk41/m6EK2YoY8Am2vWNwGHW3BcaU7XblvHsYkpxiemOJXJ+MQUxyamuHbbunmNf/eVm85sv3R9P2MnTnL0xEmu3Lx2Xsvb1vez8YJVHJuYYuMFqxZ1rHYvL/esyz3fYrNuW9/f8HxdiFZcQ78R2MVrX4p+PDN3NDqm19DVCt7lUm7W5Z6vU3e5LOpL0Yj4LHAdsB44Avwp0AOQmburty3eC+ykctvi7ZnZsKktdEmav7kKveE19My8tcH+BO5aYDZJUov4S1FJKoSFLkmFsNAlqRAWuiQVwkKXpEJY6JJUCAtdkgphoUtSISx0SSqEhS5JhbDQJakQFrokFcJCl6RCWOiSVAgLXZIKYaFLUiEsdEkqhIUuSYWw0CWpEBa6JBXCQpekQljoklQIC12SCmGhS1IhLHRJKoSFLkmFsNAlqRAWuiQVwkKXpEJY6JJUCAtdkgrRVKFHxM6IOBARByPi7jr7ByLiXyPimxGxPyJub31USdJcGhZ6RHQB9wHXA9uBWyNi+4xhdwHfzcw3A9cBfxMRK1ucVZI0h2Zm6DuAg5l5KDNPAg8CN88Yk8AFERHAauBlYLqlSSVJc2qm0C8Bnq1ZH6luq3UvcDlwGPg28P7MPDXzQBFxR0QMR8Tw6OjoAiNLkuppptCjzracsf5O4AngYuDngXsjYs1ZL8rck5lDmTk0ODg4z6iSpLk0U+gjwOaa9U1UZuK1bgceyoqDwFPAz7YmoiSpGc0U+mPAZRGxtfpF5y3AwzPGPAO8DSAiNgA/AxxqZVBJ0ty6Gw3IzOmI2AU8CnQBD2Tm/oi4s7p/N/BR4FMR8W0ql2g+lJkvtjG3JGmGhoUOkJmPAI/M2La7Zvkw8I7WRpMkzYe/FJWkQljoklQIC12SCmGhS1IhLHRJKoSFLkmFsNAlqRAWuiQVwkKXpEJY6JJUCAtdkgphoUtSISx0SSqEhS5JhbDQJakQFrokFcJCl6RCWOiSVAgLXZIKYaFLUiEsdEkqhIUuSYWw0CWpEBa6JBXCQpekQljoklQIC12SCmGhS1IhLHRJKoSFLkmFaKrQI2JnRByIiIMRcfcsY66LiCciYn9E/FdrY0qSGuluNCAiuoD7gLcDI8BjEfFwZn63Zsxa4BPAzsx8JiJ+qk15JUmzaGaGvgM4mJmHMvMk8CBw84wx7wEeysxnADLzhdbGlCQ10kyhXwI8W7M+Ut1W643AhRHxlYjYFxG/U+9AEXFHRAxHxPDo6OjCEkuS6mqm0KPOtpyx3g1cCdwIvBP444h441kvytyTmUOZOTQ4ODjvsJKk2TW8hk5lRr65Zn0TcLjOmBcz8xXglYj4b+DNwPdbklKS1FAzM/THgMsiYmtErARuAR6eMeYLwC9HRHdE9ANXAU+2NqokaS4NZ+iZOR0Ru4BHgS7ggczcHxF3VvfvzswnI+LfgG8Bp4D7M/M77QwuSfpJkTnzcvjSGBoayuHh4Y68tySdqyJiX2YO1dvnL0UlqRAWuiQVwkKXpEJY6JJUCAtdkgphoUtSISx0SSqEhS5JhbDQJakQFrokFcJCl6RCWOiSVAgLXZIKYaFLUiEsdEkqhIUuSYWw0CWpEBa6JBXCQpekQljoklQIC12SCmGhS1IhLHRJKoSFLkmFsNAlqRAWuiQVwkKXpEJY6JJUCAtdkgphoUtSISx0SSpEU4UeETsj4kBEHIyIu+cY94sR8WpE/GbrIkqSmtGw0COiC7gPuB7YDtwaEdtnGfdXwKOtDilJaqyZGfoO4GBmHsrMk8CDwM11xr0P+DzwQgvzSZKa1EyhXwI8W7M+Ut12RkRcAvwGsHuuA0XEHRExHBHDo6Oj880qSZpDM4UedbbljPWPAR/KzFfnOlBm7snMocwcGhwcbDKiJKkZ3U2MGQE216xvAg7PGDMEPBgRAOuBGyJiOjP/pRUhJUmNNVPojwGXRcRW4DngFuA9tQMyc+vp5Yj4FPBFy1ySllbDQs/M6YjYReXulS7ggczcHxF3VvfPed1ckrQ0mpmhk5mPAI/M2Fa3yDPzvYuPJUmaL38pKkmFsNAlqRAWuiQVwkKXpEJY6JJUCAtdkgphoUtSISx0SSqEhS5JhbDQJakQFrokFcJCl6RCWOiSVAgLXZIKYaFLUiEsdEkqhIUuSYWw0CWpEBa6JBXCQpekQljoklQIC12SCmGhS1IhLHRJKoSFLkmFsNAlqRAWuiQVwkKXpEJY6JJUCAtdkgrRVKFHxM6IOBARByPi7jr7fysivlX95+sR8ebWR5UkzaVhoUdEF3AfcD2wHbg1IrbPGPYU8NbMfBPwUWBPq4NKkubWzAx9B3AwMw9l5kngQeDm2gGZ+fXMPFpd3Qtsam1MSVIjzRT6JcCzNesj1W2z+X3gS/V2RMQdETEcEcOjo6PNp5QkNdRMoUedbVl3YMSvUin0D9Xbn5l7MnMoM4cGBwebTylJaqi7iTEjwOaa9U3A4ZmDIuJNwP3A9Zn5UmviSZKa1cwM/THgsojYGhErgVuAh2sHRMRPAw8Bt2Xm91sfU5LUSMMZemZOR8Qu4FGgC3ggM/dHxJ3V/buBPwHWAZ+ICIDpzBxqX2xJ0kyRWfdyeNsNDQ3l8PBwR95bks5VEbFvtgmzvxSVpEJY6JJUCAtdkgphoUtSISx0SSqEhS5JhbDQJakQFrokFcJCl6RCWOiSVAgLXZIKYaFLUiEsdEkqhIUuSYWw0CWpEBa6JBXCQpekQljoklQIC12SCmGhS1IhLHRJKoSFLkmFsNAlqRAWuiQVwkKXpEJY6JJUCAtdkgphoUtSISx0SSqEhS5JhbDQJakQ3c0MioidwD1AF3B/Zv7ljP1R3X8DcAJ4b2Y+3uKsPDV6nK/+4CWOjE3StQICmD4FGwZ62XJhHz88OnHWvuWwbL72Zd0w0Mu129axdXD1rOfHzNfXG1+7XTpXRWbOPSCiC/g+8HZgBHgMuDUzv1sz5gbgfVQK/Srgnsy8aq7jDg0N5fDwcNNBnxo9zueGR1jb18Pk1DR7D71MAm/ZdhGvTL7KN54+yo4tF9K/suvMvss3ruZ7Pzre0WXztS/rT697Hccnpzk2McU1l17E1w69fNb5MfP1q7q7zxq/urf7zHHePbTJUteyFhH7MnOo7r4mCv2XgD/LzHdW1z8MkJl/UTPmk8BXMvOz1fUDwHWZ+fxsx51voX9679O8MjnNmr4e9h56iR9PnyKAlT0rIGF8Yoo1/T2QnNn33NgJLh7o7+iy+dqX9eqt64DK+ncOj3HFxQNnnR8zX3/11nVnjT9tfGKK1/V2c9vVb2j6vJSW2lyF3sw19EuAZ2vWR6rb5juGiLgjIoYjYnh0dLSJt37NkbFJVvdWrhCNT07R272CVT0rGJ+YYnxyijV93WeWT+8bO9H5ZfO1L+tpq3u7eX5sou75MfP19cbXHufI2OS8zktpOWnmGnrU2TZzWt/MGDJzD7AHKjP0Jt77jA0DvRyvztDX9PYwWZ2BrenrOWuGeXrfQH9Px5fN18asVccnp3n9QF/d8+Os19cZX3ucDQO98zktpWWlmRn6CLC5Zn0TcHgBYxbl2m3rODYxxfjEFJeu72fsxEmOnjjJtvX9bLxgFccmpth4waqf2Hfl5rUdXzZf+7KeymR8Yqpy7fvKTXXPj5mvrze+9jjXblvXytNWWlLNXEPvpvKl6NuA56h8KfqezNxfM+ZGYBevfSn68czcMddx53sNHbzL5XzN510u0msW9aVo9QA3AB+jctviA5n55xFxJ0Bm7q7etngvsJPKbYu3Z+acbb2QQpek891chd7UfeiZ+QjwyIxtu2uWE7hrMSElSYvjL0UlqRAWuiQVwkKXpEJY6JJUiKbucmnLG0eMAk935M0XZz3wYqdDLDE/c/nOt88L5+5nfkNmDtbb0bFCP1dFxPBstwyVys9cvvPt80KZn9lLLpJUCAtdkgphoc/fnk4H6AA/c/nOt88LBX5mr6FLUiGcoUtSISx0SSqEhb4IEfHBiMiIWN/pLO0UEX8dEd+LiG9FxD9HxNpOZ2qXiNgZEQci4mBE3N3pPO0WEZsj4j8j4smI2B8R7+90pqUSEV0R8b8R8cVOZ2kVC32BImIzlQdnP9PpLEvgy8AVmfkmKn8b/8MdztMW1Qei3wdcD2wHbo2I7Z1N1XbTwB9k5uXA1cBd58FnPu39wJOdDtFKFvrC/R3wh9R51F5pMvPfM3O6urqXyhOpSrQDOJiZhzLzJPAgcHOHM7VVZj6fmY9Xl/+PSsGd9Tzg0kTEJuBG4P5OZ2klC30BIuJdwHOZ+c1OZ+mA3wO+1OkQbdLUw85LFRFbgF8A/qfDUZbCx6hMyE51OEdLNfWAi/NRRPwHsLHOro8AfwS8Y2kTtddcnzczv1Ad8xEq/4v+maXMtoSaeth5iSJiNfB54AOZOd7pPO0UETcBL2Tmvoi4rsNxWspCn0Vm/nq97RHxc8BW4JuVJ++xCXg8InZk5o+WMGJLzfZ5T4uI3wVuAt6W5f54oe0PO1+OIqKHSpl/JjMf6nSeJXAN8K7qozV7gTUR8Y+Z+dsdzrVo/rBokSLih8BQZp6Lf7WtKRGxE/hb4K2ZOdrpPO3SzAPRS1N9HvDfAy9n5gc6HGfJVWfoH8zMmzocpSW8hq5m3AtcAHw5Ip6IiN2NXnAuqn7xuwt4lMqXg/9UcplXXQPcBvxa9d/tE9WZq85BztAlqRDO0CWpEBa6JBXCQpekQljoklQIC12SCmGhS1IhLHRJKsT/A0qhAAdy8SDtAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_data = numpy.where(y_data >= 0.5, 1, 0)\n",
    "pyplot.scatter(x_data, y_data, alpha=0.4);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic loss function\n",
    "\n",
    "To use gradient descent, we need a loss function that we can optimize with respect to the parameters. \n",
    "If we were to use a mean-square-error loss function, like in linear regression, we'd run into a few problems. Let's discuss.\n",
    "\n",
    "Taking the derivatives to optimize would involve $\\sigma^{\\prime}(z)$, so let's look at that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{e^{- z}}{\\left(1 + e^{- z}\\right)^{2}}$"
      ],
      "text/plain": [
       "exp(-z)/(1 + exp(-z))**2"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lprime = logistic.diff(z)\n",
    "lprime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAEfCAYAAAAQiIIwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzJ0lEQVR4nO3deXhb1Z0+8PdKsrxvsi3HtrxGXhInjuMtCyUpkDgLNNPQFgLDQJuhmbR04DctMEyZmS7TKbTDdGibDqnL0sIDyXQBTCELW0ggJHEcZ3cWE+/yvjveJZ3fH17wGsuJpHslvZ8nemzpnit9feXo9b333HMkIQSIiIiUSiV3AURERNfCoCIiIkVjUBERkaIxqIiISNEYVEREpGgMKiIiUjQGFRERKRqDioiIFI1BRUREiqaRuwCi2UiS5APgDgA3A4gG0AfgHIB3hBDn5ayNiBxP4hBKpGSSJP0QwJcAfATgBIAmAD4AUgDcMvL994QQZ2QqkYgcjEFFiiZJ0u1CiHeusVwPIE4IUezEsojIiRhU5BIkSboZwKdCCMu4x7KEECUylkVETsDOFOQq9gP4UJKkyHGPPS9XMUTkPLMFleCNNyXcMjMzff/617+uSktLa/j0009HH1sqd12Tb+vXr5e9Bt54c7HbrNjrj1yCJEm44447kJqairvvvhtbt26FJElylzVFS0uL3CUQuR0e+iOXMHouNTk5GZ988gkOHTqEM2fY0Y/IE8zWmcKm3TIiOVRXVyMuLk7uMibIyclBcTE7IBLNwayHRrhHRYr2k5/8BG1tbdMui4uLw4cffoi3337byVURkTPxHBUp2uLFi/GlL30JPj4+yMrKQkREBPr7+1FWVoZTp05hzZo1+P73vy93mUTkQAwqUrQ///nPOHz4MH7+859Dr9ejvr4eQUFBuO+++1BQUABfX1+5SyQiB2NQkaKdOHECVVVVePXVV3HgwIEJy/r6+mYNqn379uGRRx6BxWLBgw8+iCeeeGLC8ldffRU/+9nPAAABAQF47rnnsGTJEgBAQkICAgMDoVarodFoeO6JSCYMKlK07du3Y/369SgvL0dOTs7Y40IISJKE8vLyGde1WCx46KGH8N5778FgMCA3NxebNm3CwoULx9okJibi4MGDCA0Nxd69e7Ft2zYcO3ZsbPmBAwcQHh7umB+OiGzCzhSkaA8//DAuXLiArVu3ory8fOxWUVFxzZACgKKiIhiNRiQlJUGr1WLLli0oLCyc0GblypUIDQ0FACxfvhy1tbUO+1mI6PowqMglPPfcc3Nex2QyITY2duy+wWCAyWSasf0LL7yADRs2jN2XJAn5+fnIzs5GQUHBjOs9+9zzyFy9ARlrv4qaxla8fKQSlxq6YLXy6g4ie+ChP3Jb010jONNoFgcOHMALL7yATz75ZOyxw4cPIzo6Gk1NTVi7di3S0tKwatWqCeu9cbIWewdS0LHiOwCAjnOn8e+F5xEd7INv32LEfcvj7fgTEXkm7lGR2zIYDKipqRm7X1tbi+jo6Cntzpw5gwcffBCFhYUICwsbe3y0rV6vx+bNm1FUVDRhPVNHH574y1mE+nnh1/csxWsPLkOyPhDHn7wN8WH++Nm+i2juHnDQT0fkORhU5LZyc3NRVlaGiooKDA4OYvfu3di0adOENtXV1bjzzjvxyiuvICUlZezxnp4edHd3j33/7rvvYtGiRRPW/fm+iwCA/74rE19aEo2VxnD4eKkQEeiD//jyIvQPWfDTPRcc/FMSuT8e+iO3pdFosGPHDqxbtw4WiwVbt25Feno6du7cCWC4R+GPf/xjtLa24tvf/vbYOsXFxWhsbMTmzZsBAGazGffeey/Wr18/9twnq9tReKoO/3irETEhU7vIG/UB2L56Pn794Wf4Wo4BK+ez5yDR9eJYf0RzJITAV3ceQXVbLz569Ivw9/78773xY/31D1mQ/z+HoFFL2PvIzfDWqOUqmUjJONYfkb29c7YeJ6ra8Wh+yoSQmszHS40f/006ypt78LtD1+5KT0QzY1ARzUH/kAVP772IBVFB+Gp27Kztv5iqx8bF8/DrDz9DdWuvEyokcj8MKqI5eOlwJWrb+/Cvty+AWmXbxI3/fkc6NCoJ//7WuWm7zBPRtTGoiGzUcnUAvznwGdYs0OMmo+2dI+YF++C7+an46FIz9p1rcGCFRO6JQUVko/957zL6hyz4l40L5rzuAyvisTAqCG+fqYeFI1YQzQmDisgGpvZe7D/fgPuWx2N+RMCc19eoVfjmqiS8c7YeJdXtDqiQyH0xqIhs8MfiWnT2DeGbNyde93OsWaCHVq3Cfh7+I5oTBhWRDd4rbcQSQwhiQv2u+zkCfbyw0hiGd0sb2amCaA4YVESzMHX0obS+C2sWRt7wc+UvnIfqtl5cbOi2Q2VEnoFBRTSLDy40AgDW2iGo1izUQ5KAd8833vBzEXkKBhXRLN4rbURSuP91daKYTB/og6y4UOw/z/NURLZiUBFdQ3f/EI6Wt9rlsN+odemRKK3vQk0bR6ogsgWDiugaDl1uwZBFYM0C+wVV/sJ5AIb31Ihodgwqomt4r7QBoX5eyI4PtdtzJoT7IzUykIf/iGzEoCKawZDFig8vNuHWtEibx/WzVX56JI5XtqGtZ9Cuz0vkjhhURDMormxHV78Zaxfq7f7c69LnwSqA9y/w8B/RbBhURDN4/0IjtGoVbk6OsPtzp0cHITrYh93UiWzAoCKahhAC719oxEpj2DUnR7xekiQhP30ePi5rRu+g2e7PT+ROGFRE0/is6SqqWnvtcpHvTPLTIzFgtuLQ5WaHvQaRO2BQEU3j3ZGu47elOS6o8hJ0CPHzwn4e/iO6JgYV0TTev9CIDEMw5gX7OOw1NGoVbkuLxAcXGjFksTrsdYhcHYOKaJKm7n6cqumw60W+M8lPj0RXvxlFFW0Ofy0iV8WgIprkwMUmCGGfQWhnsyo5Aj5eKl78S3QNDCqiSd4rbUJMiC/S5gU6/LV8tWqsSo7Au+c5RxXRTBhUROP0DVrwyWfNWLswEpJk39EoZrJhcRT0Qd64xDmqiKbFoCIa50RVG2JCfHFrmv1Ho5hJTnwoztR2oqiS56mIpsOgIhrnUFkLatr7sCxJ57TXNIT6Yl6QDztUEM2AQUU0zvHKNiwxBMNbo3baa0qShNxEHY5XtvE8FdE0GFREI/oGLThb24ncBOftTY3KSwhFY9cAatv7nP7aRErHoCIacaqmA2arkCWochOHX5OH/4imYlARjSiubIMkAVlx9psk0VYp+kAE+WhwnB0qiKZgUBGNKKpsQ2pkIIL9vJz+2iqVhJwEHXv+EU2DQUUEwGyxoqSqXZbDfqNyE3Qob+5By9UB2WogUiIGFRGAiw3d6Bm0ICfB+Yf9RuUlDr92MfeqiCZgUBEBY+eG5NyjWhwTAm+NCscr22WrgUiJGFTk1vbt24fU1FQYjUY8/fTTU5a/+uqryMjIwH8W7Ia6vwPNVZdtXtfetBoVMmND2KGCaBIGFbkti8WChx56CHv37kVpaSl27dqF0tLSCW0SExPx0UcfITQlF9lxIdi2bZvN6zpCXqIO5+u60DPA6emJRjGoyG0VFRXBaDQiKSkJWq0WW7ZsQWFh4YQ2K1euRLfwRnP3ANZkJqG2ttbmdR0hN0EHi1WgpJqH/4hGMajIbZlMJsTGxo7dNxgMMJlMU9qNnhOqKHoPGzZsmNO69pYVHwqVBBznhb9EYzRyF0DkKNONmzfd1B3HK9rgrwHe+MNOHP7k4zmtCwAFBQUoKCgAADQ3N99IyQjw1mBhdBCvpyIah3tU5LYMBgNqamrG7tfW1iI6OnpKu08u16On+hzeKnwTYWFhc1oXALZt24bi4mIUFxcjIiLihuvOTdDhZHUHBs3WG34uInfAoCK3lZubi7KyMlRUVGBwcBC7d+/Gpk2bJrQ5c6kcpi4zttyWi5SUlDmt6yh5CToMmK04a+p0yusRKR2DityWRqPBjh07sG7dOixYsAB33XUX0tPTsXPnTuzcuRMA8K+/fAkA8PZLzyIzMxM5OTnXXNcZckau5eKFv0TDpFnmv+HkOOTWfvJ2KV4+WoWzP8y3yxxUOTk5KC4uvuHnufWZj5AU4Y/nH8i94eciUrjpT/6Owz0q8mjHq9qRaQhx6kSJtshN0OF4ZTusVv6tSMSgIo/VO2jGeVMnchPlG99vJrmJOnT2DaGs6arcpRDJjkFFHutU9fBEiTkyju83k7yRmthNnYhBRR7seGW7bBMlziZW54vIIG9e+EsEBhV5sOOjEyX6On+ixNlI0vBEiscr26a9+JjIkzCoyCOZLVaUVLcjL1F5h/1G5SXoUN/ZD1NHn9ylEMmKQUUe6UJ9N3oHLYo8PzUqL1GHJYZgnKzukLsUIlkxqMgjldZ3IiUyADnxIXKXMqNkfQDKmq7iRBVHUifPxqAij/RxWQt6BiyIDvGTu5QZadQqZBiCcZJTfpCHY1CRRzpV04HMuBC5y5jV0rhQnK/rQv+QRe5SiGTDoCKP09Tdj9r2PiyNDZG7lFktjQ2B2Spwvo4D1JLnYlCRxzk10jlhqQvsUY3u9bFDBXkyBhV5nJM1HfBSS0iPDpa7lFnpA30QE+KLkzUdcpdCJBsGFXmck9XtWBgVBB8vZQ1EO5OlcSFje4FEnohBRR7FYhU4U9uJTBc4PzVqaVwoTB19aOzql7sUIlkwqMijXG4cvtB3qQLH95vJUp6nIg/HoCKPctKFOlKMSo8OglatwskaXk9FnolBRR7lZHU7dP5axOmUe6HvZN4aNRZEB/E8FXksBhV5lFM1HciMDYEkzTr7taIsjQ3BmdpOmC1WuUshcjoGFXmM0RlzXeFC38mWxoWgb8iCS43dcpdC5HQMKvIYZ2o7AMClOlKMGp3ckR0qyBMxqMhjnKzugCQBGbHKv9B3MkOoL8IDtAwq8kgMKvIYp2o6YIwIQJCP8mb0nY0kSciMDcEp9vwjD8SgIo8ghMDJ6naX6pY+2dK4UFxp7kFn75DcpRA5FYOKPEJVay/ae4dc8vzUqNFOIKdGzrUReQoGFXmE0YtlXXmPKiM2BJIETqRIHodBRR7hVHUH/LVqJOsD5S7lugV4a5AaGcgOFeRxGFTkEU7WdCDDEAK1yrUu9J1suENFB4QQcpdC5DQMKnJ7/UMWlNZ1ufRhv1FL40LQ2TeEipYeuUshchoGFbm9c6ZOmK3CpTtSjFrKC3/JAzGoyO2Nfqi70hxUMzFGBCDQW8OR1MmjMKjI7Z2q6YAh1BcRgd5yl3LDVCoJdyyJRmPngNylEDkNg4rc3vCFvq5/2G9UeIAWH1xsRM+AWe5SiJyCQUVuraGzH3Wd/S45YvpMsuJCYRXAaV74Sx6CQUVubXRsvEw36PE3ilPTk6dhUJFbO1nTAa1ahfToILlLsZsQPy2SIvw5QgV5DAYVubUzlyowVHMa6WmpePrpp6csv3jxIlasWAFvb28888wzE5YlJCRg8eLFyMzMRE5OjrNKtklWXChKqnnhL3kGBhW5rf7BIRypHcDm/FUoLS3Frl27UFpaOqGNTqfDr371Kzz66KPTPseBAwdw6tQpFBcXO6Nkm2XFhaKtZxBVrb1yl0LkcAwqclt/fv8ooNHilsUJ0Gq12LJlCwoLCye00ev1yM3NhZeXa81RlRUfAgAo4eE/8gAMKnJbxRUtAIDs+OGu6QaDASaTyeb1JUlCfn4+srOzUVBQ4JAar1eyPhAB3hoGFXkEjdwFEDlK5VUVvC19iA7xHXtMkmwflPbw4cOIjo5GU1MT1q5di7S0NKxatWpKu4KCgrEga25uvvHCbaBWDc/4W1LV4ZTXI5IT96jIbdX2a+HVVfv5/dpaREdH27z+aFu9Xo/NmzejqKho2nbbtm1DcXExiouLERERcWNFz8HSuBBcbOjihb/k9hhU5JaauvrR0mdFT9VZVFRUYHBwELt378amTZtsWr+npwfd3d1j37/77rtYtGiRI0ueM174S56Ch/7ILY2eu3n063di3bp1sFgs2Lp1K9LT07Fz504AwPbt29HQ0ICcnBx0dXVBpVLh2WefRWlpKVpaWrB582YAgNlsxr333ov169fL9vNMZ/yFvyvnh8tbDJEDMajILZ2oaodWrcKDX83HQ1tun7Bs+/btY9/PmzcPtbW1k1dHUFAQTp8+7fA6b8Tohb8lVexQQe6Nh/7ILZVUd2BRTBC8NWq5S3GorLhQnOSMv+TmGFTkdgbNVpw1dSLLjUZMn8nohb+VvPCX3BiDitzO+bpODJqtY9dPubOxC395+I/cGIOK3E7JyKjiWR4QVLzwlzwBg4rcTklVO2JCfBEZ5CN3KQ43euEvp/wgd8agIrdTUt0+1nXbE2Txwl9ycwwqciv1nX2o7+z3iPNTo5bG88Jfcm8MKnIro2PfeUKPv1FZscM/Kw//kbtiUJFbOVHVDm+NCgui3GdG39kE+3lhPi/8JTfGoCK3UlLdjgxDMLQaz/rV5oW/5M48638zubX+IQvO13V6RLf0yZbywl9yYwwqchvn67oQHuCNrNgQuUtxuqy4EOQl6HCWHSrIDTGoyG0UVbR5XI+/USmRgbjc1I1PPmuRuxQiu2NQkdsoqmiFUR+A8ED3v9B3MpVKQm6CDscq2uQuhcjuGFTkFixWgeLKduQl6uQuRTbLEnWoau1FQ2e/3KUQ2RWDitzChfoudA+YscyDg2p5UhgA4FhFq8yVENkXg4rcQtHIIa/cBM8NqgVRQQj00eBoOQ//kXthUJFbKKpoQ6zOF9EhvnKXIhv12Hkq7lGRe2FQkcsTQqCosg15CWFylyK7ZYk6lDf3oKmb56nIfTCoyOVdab6Ktp5Bjz4/NWrZyHmqIvb+IzfCoCKXV1QxPMadJ/f4G7UoOgj+WjWO8TwVuREGFbm8oopW6AO9ER/mJ3cpstOoVcjmeSpyMwwqcmlCCByraENeog6SJMldjiIsS9ThcuPw4VAid8CgIpdW2z48USIP+31uedLwtijiXhW5CQYVubTRTgMMqs8tjgmBj5eKwymR22BQkUsrqmhDsK8XUvSBcpeiGFqNCllxoexQQW6DQUUuraiyDbkJOqhUPD813rLEMFxo6EJn75DcpRDdMAYVuaymrn5UtPTw+qlpLEvSQQjgeCX3qsj1MajIZRVV8vzUTDJjQ6DVqNhNndwCg4pcVlFFG/y0aqRHB8ldiuL4eKmRGRvCDhXkFhhU5LKKKtqQHR8KjZq/xtNZnqjDOVMnuvt5nopcG/+Hk0tquzqAS43dPD91DcuSwmAVQHFVu9ylEN0QBhW5pKMVbQj19cLK+RwxfSZZcaHwUkvspk4uj0FFLunjsmYMWQQWG0LkLkWxfLVqZBhC2KGCXB6DilyOEAKHLrdgxfwwePH81DUtS9ThbG0negfNcpdCdN34v5xczpXmHpg6+rAqJULuUhRvWVIYzFaBkqoOuUshum4MKnI5H5c1AwBW2xBU+/btQ2pqKoxGI55++ukpyy9evIgVK1bA29sbzzzzzJzWdQXZ8aFQqyQe/iOXxqAil3PocjMSwvwQq7v2/FMWiwUPPfQQ9u7di9LSUuzatQulpaUT2uh0OvzqV7/Co48+Oud1XUGAtwY3zQ9HTVuv3KUQXTcGFbmUAbMFR8vbbDrsV1RUBKPRiKSkJGi1WmzZsgWFhYUT2uj1euTm5sLLy2vO67qKDEMw/nqmnuP+kctiUJFLOVHZjr4hC25Onj2oTCYTYmNjx+4bDAaYTCabXmcu6xYUFCAnJwc5OTlobm626fmd6ZY0PSxWgYNlyquNyBYMKnIpB8uaoVFJWGHD9VNCiCmP2ToL8FzW3bZtG4qLi1FcXIyICOV18MiMDYHOX4sDF5vkLoXoujCoyKV8fLkF2fGhCPDWzNrWYDCgpqZm7H5tbS2io6Ntep0bWVdp1CoJX0yJwEeXmmCxTg1gIqVjUJHLaO4eQGl9l83d0nNzc1FWVoaKigoMDg5i9+7d2LRpk8PXVaJb0vRo7x3CqRoOp0SuZ/Y/S4kU4pPPhs+xrLLh/BQAaDQa7NixA+vWrYPFYsHWrVuRnp6OnTt3AgC2b9+OhoYG5OTkoKurCyqVCs8++yxKS0sRFBQ07bqualVKBNQqCR9ebEJ2PMdHJNciTXcsfhweJyDF+Kf/O4WDl5tR/OQaxc7om5OTg+LiYrnLmNZdvz2C7n4z9j5ys9ylEI03639mHvojl2C1Cnxc1owvGMMVG1JKd2uaHhfqu1Df2Sd3KURzwqAil3ChoQstVwc5bNINuDVNDwA4cJHd1Mm1MKjIJRy63AIAWJUcLnMlritZH4CYEF98eLFR7lKI5oRBRS7h47JmpM0LhD7IR+5SXJYkSbhtgR6HP2tF/5BF7nKIbMagIsXrHTSjuLKdh/3s4JY0PfqGLDhazkFqyXUwqEjxjpa3YtBitblbOs1sRVIYfLxUHKWCXAqDihTv0OUW+HipkJMQKncpLs/HS42b5ofjw0tN0w4TRaREDCpSNCEELtR34ms5sfDxUstdjlu4JU2PmrY+fNZ0Ve5SiGzCoCJFO1/XhWMV7UiPCpK7FLdxy0g39Q95+I9cBIOKFG3fuQaoVRLy0+fJXYrbiAnxRdq8QAYVuQwGFSmWEAJ7ztVjeZIOOn+t3OW4lVvT9CiuakdnHydTJOVjUJFilTVdRXlzD9YvipK7FLdz68hkih9zMkVyAQwqUqw9Z+shScC69Ei5S3E7S+NCEeLnxcN/5BIYVKRYe882IDdeB30gR6OwN7VKwuqUCHx0qZmTKZLiMahIka40X8Wlxm5sWMxOFI5ya5oebT2DOF3bIXcpRNfEoCJF2neuAQCwfhGDylFWp0QgIsAbR69wOCVSNgYVKdLec/VYGheCqGBfuUtxWyF+WqRFBeK1ompYefiPFIxBRYpT3dqLc6YubGRvP4fbvDQGte19KK5ql7sUohkxqEhx9p6rB8DDfs6wLn0e/LRqvHGyVu5SiGbEoCLF2XuuAYtjghGr85O7FLfn763B+vR5ePtMPeeoIsViUJGi1HX04VRNB/emnOjOLAO6+814/wJn/iVlYlCRooz29tvAoHKaFfPDMC/IB2+UmOQuhWhaDCpSlDO1HVidEoGkiAC5S/EYapWELy+NwUeXm9FydUDucoimYFCRYlxs6MKbp+rwBWO43KV4nDuzYmCxCrx1qk7uUoimYFCRYrx2rBpatQpfyTbIXYrHSYkMxKKYILzO3n+kQAwqUoTeQTPeKDFh4+J5nNJDJncuNeCcqQuXG7vlLoVoAgYVKcLbp+vRPWDGvcvi5S7FY23KjIZaJeF1dqoghWFQkSK8WlQNoz4AuQmhcpfiscIDvLE6JQJvnjRxRHVSFAYVye58XSdO13Tg3rw4SJIkdzke7c6sGDR09eNoOQeqJeVgUJHsXjtWDW+NCl/JYicKua1ZEIlAHw3+UsJOFaQcDCqS1dUBM948acIdGdEI9vOSuxyP5+Olxu2Lo7DvXAN6Bsxyl0MEgEFFMnvrVB16Bi24d1mc3KXQiDuzDOgdtGDv2Xq5SyECwKAimb1WVIW0eYHIiguRuxQakRMfio2L52HX8RoIwU4VJD8GFcnmTG0Hzpm6cO8ydqJQEpVKwsr54ThR1Y6j5W1yl0PEoCL5vHasGr5eanx5aYzcpdAkX802IDxAi50Hr8hdChGDiuTR3T+Et07XYdOSaAT5OK4Txb59+5Camgqj0Yinn356ynIhBB5++GEYjUZkZGSgpKRkbFlCQgIWL16MzMxM5OTkOKxGJfLxUuMbNyXi4OVmnK/rlLsc8nAMKpLFm6fq0OvgThQWiwUPPfQQ9u7di9LSUuzatQulpaUT2uzduxdlZWUoKytDQUEBvvWtb01YfuDAAZw6dQrFxcUOq1Op7lsejwBvDX57sFzuUsjDMajI6YQQeO1YNdKjg5BhCHbY6xQVFcFoNCIpKQlarRZbtmxBYWHhhDaFhYW4//77IUkSli9fjo6ODtTXs7cbAAT7euHeZXF4+0wdqlt75S6HPBiDipzu0OVmqCRg26okh3aiMJlMiI2NHbtvMBhgMplsbiNJEvLz85GdnY2CggKH1alkW29KhFol4Xcfc6+K5KORuwDyLEIIPPPuZXT0Djl8uvnpulZPDsZrtTl8+DCio6PR1NSEtWvXIi0tDatWrZrSvqCgYCzImpub7VG6YswL9sGdSw34Y3ENHlmTjPAAb7lLIg/EPSpyqn3nGnDW1In/tyYZ3hq1Q1/LYDCgpqZm7H5tbS2io6NtbjP6Va/XY/PmzSgqKpr2dbZt24bi4mIUFxcjIiLC3j+G7LatTsKgxYo/fFopdynkoRhU5DQWq8Az716CUR+AO50wrl9ubi7KyspQUVGBwcFB7N69G5s2bZrQZtOmTXj55ZchhMDRo0cRHByMqKgo9PT0oLt7eF6mnp4evPvuu1i0aJHDa1ai+REByF8YiZePVOEqh1UiGfDQHznN6yW1uNLcg+f+NgtqleMv8NVoNNixYwfWrVsHi8WCrVu3Ij09HTt37gQAbN++HRs3bsSePXtgNBrh5+eHl156CQDQ2NiIzZs3AwDMZjPuvfderF+/3uE1K9X21fOx/3wjdhdV48Gbk+QuhzyMNMsQKRw/hexiwGzBrc8chM5fi7e+c5PbjkSRk5Pjtl3ZtxQcQWVLLw49fgu0Gh6MIbuZ9cOAv23kFLuOVcPU0YfH1qW6bUi5u+2r56Ohqx9vnuQUIORcDCpyuN5BM3Yc+AzLk3S4OTlc7nLoOq1OicDmpTH4w6dV6Bu0yF0OeRAGFTncS4cr0XJ1EI+tS+PelAuTJAn35MXhfH0Xdhwok7sc8iAMKnKozt4h7Dx4BWsW6JEdHyp3OXSD8hJ1+EqWAQWHyvFZ01W5yyEPwaAih9p56AquDpjxvfxUuUshO/mXjWnw9VLj3948x/mqyCkYVOQwTd39eOlwBTYticaCqCC5yyE7CQ/wxuPr03CkvBVvna6TuxzyAAwqcpj/3n8ZGTEh+Kc1KXKXQnZ2T14clhiC8R9vX0BX/5Dc5ZCbY1CRQ+w7V4//K67BivlhSAj3l7scsjO1SsJPvrwYbT0D+MW7l+Uuh9wcg4rsrrl7AN9/4xwWxwTjO7ca5S6HHGSxIRh/tzweLx+pxDkTJ1ckx2FQkV0JIfD9N87i6oAZv7hrCbzU/BVzZ9/NT4XO3xtPvnkOFis7VpBj8FOE7OovJSa8V9qIx9elIjkyUO5yyMGCfb3wr7cvwOmaDuw+Xi13OeSmGFRkN6aOPvzorfPIS9Rh602JcpdDTvI3mdHYsGgeXi8xobyZ11aR/TGoyC6sVoHH/nQaViHw319bApUTRkcnZZAkCU/evgAVLT34h1dOcCoQsjsGFdnF7z+txKdXWvFvdyxErM5P7nLIyQyhfthxz1Jcab6Kx/50mhcCk10xqOiGfXqlBS8ersC9eXG4OzdW7nJIJiuN4XhiQxr2nmvAcwevyF0OuREGFd2Qy43d+IdXTsDXS41/Xs9BZz3dN29Owh0ZUXhm/yUcutwsdznkJhhUdN0au/rx9ReL4OOlxkvfyEWwn5fcJZHMJEnCz7+agWR9IP5x10nUtPXKXRK5AQYVXZerA2Z846Xj6Ogbwktfz4UhlOelaJifVoPf/l02hBD44V/Po6N3UO6SyMUxqGjOhixWfPvVElxq7MZv/jYLi2KC5S6JFCYh3B/PP5CDo1dacc/vjqH16oDcJZELY1DRnFisAv/xdikOXW7GT768CLek6uUuiRQqLzEM/3tfNsqbr2JLwVE0dfXLXRK5KAYV2ax/yIKHXi3B6yW1+NfbF+CevDi5SyKFW50Sgd9/Iw+mjj7cXXAUdR19cpdELohBRTZp6xnE3z5/DPtLG/BPa1Px4M1JcpdELmLF/DC88vd5aOkewF2/PcIOFjRnDCqaVXVrL77y3Kc4a+rE/96bhb//AodHornJjtfh1W8uQ3e/GXf99giHWqI5YVDRNZ2u6cCdzx1Ge+8gXntwGTYsjpK7JHJRGYYQ7N62HINmK35QeB4fXmyUuyRyEQwqmpYQAm+U1OKe3x2Fr1aNv3xrJXISdHKXRS5uQVQQ3nzoJrT0DGDr74vx/TfOoodjA9IspFnG5OKAXR6osasfT75xDu9faMSW3Fh8Lz8VEYHecpflEnJyclBcXCx3GYo3YLbgF+9dRsGhcsTr/PCLuzORFRcqd1kkj1mHs2FQ0RghBP7veA3+c88FDJqteDQ/FVu/kAg1R0K3GYNqbo6Vt+K7fzyN+s4+PHSLEQ/flszJNj0Pg4psU93aiydeP4NPr7RiWaIOP/tKBhLC/eUuy+UwqOauu38IP/prKf58ohaLY4LxP3cvgVHPSTc9CIOKrq1/yILXjlXj5/svQqNS4fsbF2BLbiznk7pODKrrt+9cPf7l9bOICfXF8sQwfHNVEiKDfOQuixyPQUXT6x0047Vj1Sg4VA6dvxYxob74yZcXISrYV+7SXBqD6sY0dffjd4fK8eLhSqglCV/LMWD76vmc48y9Mahoou7+IbxytAovfFyB1p5BrEgKwz/easSK+WGcosMOGFT2Ud3ai+cOXsFfTtTCIgS+nBmDb98yH/MjAuQujeyPQUXDnSSOV7bjzydqUN7cg+KqdnwxNQLfucXILud2xqCyr4bOfhQcKsdrRVUYMFvxpYwo3JUTh+VJOmjY6cJdMKg8WU1bD14vqcNfSmpR3dYLP60aX8qIwv0rEpDOEc8dgkHlGC1XB/DiJxW4UN+FA5eaofPXYl16JDYsisKK+WHsKejaGFSe5OqAGUUVrfikrBWffNaMQB8vlFS3Y+X8MHwly4D1i+bBT6uRu0y3xqByrP4hCz661Iy95+rxwYUmXB0wI8TPC/kLI7FxcRRWJIXB20std5k0NwwqdzY4ZMHZui58UtaCw5+1oKS6HWargLdGhbxEHTYsmofVqXrEhLCDhLMwqJynf8iCj8tasOdsPd4vbUT3gBmpkYHQ+WuRkxCK7PhQZMWHIsiHM08rHIPKXXT1D+FCXRcu1HehtL4LF+q7UdXWg6v9ZggAi6KDcZMxHDcnhyM7PhQ+/KtSFgwqeQyYLThypRUHLzXjeFUbLtR3w2IVkCQgRR+I2xboERnkg+TIAKREBiI8gCOtKAiDylUIIdDWO4j6jn5Ut/Wiuq0XVa29qGnrhY+XCu9faBprq/PXYkFUIBZGBSEvUYfseB10/loZq1euffv24ZFHHoHFYsGDDz6IJ554YsJyIQQeeeQR7NmzB35+fvj973+PrKwsm9adDoNKGXoGzDhd04HiqnYUV7XDYrHi8JXWseU6fy2S9QGYrw9AdLAPYkJ9ERPih5hQX0QGerOjhnMxqOQyZLGiZ8CMrj4zOvuG0NE3iI7eIXT0DmLAbEVlaw+augbQ1D2A5pGbJAkMmD/f5Dp/LeJ0fsiJD0WovxYLo4KwMDoI+kBvdiW3gcViQUpKCt577z0YDAbk5uZi165dWLhw4VibPXv24Ne//jX27NmDY8eO4ZFHHsGxY8dsWnc6DCplEkKgqXsAlxu7cbnxKsoau1Hd1otLDd1o7Rmc0FatkpATH4ohixVhAd4ID9AizN8bYQFa6Py1CPXTItjXC0G+Xgjy0SDQxwtaDYPtBsz6YSbLmXUhBIYDUoIYvT+6zCogJGA0P63W4WUCgFUICDHc3jrNV4uwwmodXtciBCxWAasYvpktw18t1uGbgMCgWWDIYoXFKjBkETBbrbAKoH/QggGLFUNmKwbHffXxUqGpawB9Qxb0DVnRP2RB/5AFAd4aVLX2omfQjJ4BM7r7zRgwWwEA84J80DBpCu7liTpcauxGRKA39IE+SAr3R0TQ8PeGUF8YQn0Rp/NDII+t35CioiIYjUYkJQ1P8rhlyxYUFhZOCJvCwkLcf//9kCQJy5cvR0dHB+rr61FZWTnruuQ6JElCZJAPIoN8cHNyxIRlfYMWmDr6hm/tfTB19GLQbMWF+m7UtPXiVE0H2noGYbEKaFTAyH/tCXLiQ1HR0gM/bzX8tRr4adXw99YgKsgH/ebhzw4fLzVC/bxgFYBWrYKXRgWtWgWtRoVAHw2sQkCjUsFLLUGjUsFXqwIgQa36/KZRSVBLElQj91WSBJWEz79XDd9XSRIkYOT+8GMSJEgqAGL4cQkj7UZiQhrZTpI07nuIsT+K5fzj2ClBdbGhC+uf/XjCY4YQX9ROMy31EkMwTtd2jt2fF+SNhq6BKe1iQnxhmrR+fJgvqlqnPufSuBCcrO6Y8FhmbAhO1XRMaRsT6gtT+9TnkCRgeVIYSuu64OulHvvF89WqEeitQXSID/y9NfD31iBw5Ku/twY6fy8EeHshxM8LIb5eCPHTIshHw55JTmAymRAbGzt232Aw4NixY7O2MZlMNq1L7sFXq4ZRHwCjfuaLia1Wgc6+IbT2DKC9dwhdfUPo7jejq3/4e6sAUucFonfQgp4BM3oHLbg6YEZdZz9q23vRP2RFv9mC6GBfXGjowuQDWTnxoSiuap/wWF6iDkUVbVNqUUmA1YZjXfMj/HGluWfK43kJOhRVTn3evMRQFFW029R2531ZWL/IeXPTXfPQ3/r160VLS8sNv4jZKlDT1A4/v8+HQVFJ0udv1rigHv8mDKf68B7S5DBXSRImrz6+zcjfAMN/HUgY3iUb/atiXFtp5C+PsWUj7Uf/ImltbUVERPjs+6YK1NzcjIiIiNkbKpA9am9vb0dXVxfi4+MBDL+XPT09iIuLG2tTVlaGqKgoBAQMf0hdvnwZBoMBAwMDs647vtbR/ycDAwPIzMy8obrl4qq/L65Y9+iRpJaWVuh0YbBi+GgRBCBGvp9ytGlk2ci/scdGHh174vGfi1Z8/pyjVJIEixATTuwIAGoJsIyPAwGoVIBlmlS09l9FlD78hrbBqBMnTuwXQqy/ZqPRw3Az3OwmOzvbnk/nNK5atxCs/dNPPxX5+flj93/605+Kn/70pxPabNu2Tbz22mtj91NSUkRdXZ1N607Hz8/vhuuWi6v+vrhq3UK4bu12rnu2HOIMv+S+cnNzUVZWhoqKCgwODmL37t3YtGnThDabNm3Cyy+/DCEEjh49iuDgYERFRdm0LhE5B4cpILel0WiwY8cOrFu3DhaLBVu3bkV6ejp27twJANi+fTs2btyIPXv2wGg0ws/PDy+99NI11yUi53NaUG3bts1ZL2VXrlo3wNoBYOPGjdi4ceOEx7Zv3z72vSRJ+M1vfmPzurMJD7fPcXs5uOrvi6vWDbhu7c6um9dREdkRr6MimrNZ+6rxHBURESmaXYPqT3/6E9LT06FSqab8VfnUU0/BaDQiNTUV+/fvn3b9trY2rF27FsnJyVi7di3a29unbedId999NzIzM5GZmYmEhIQZuxonJCRg8eLFyMzMRE5OjnOLnMEPf/hDxMTEjNW/Z8+eadvt27cPqampMBqNePrpp51c5VSPPfYY0tLSkJGRgc2bN6Ojo2Padkra5rNtQyEEHn74YRiNRmRkZKCkpESGKieqqanBLbfcggULFiA9PR2//OUvp7T56KOPEBwcPPY79OMf/1iGSqc32/uvxG0OAJcuXRrbnpmZmQgKCsKzzz47oY1StvvWrVuh1+uxaNGiscds/Vx26OfKLN0C56S0tFRcvHhRrF69Whw/fnzs8fPnz4uMjAzR398vysvLRVJSkjCbzVPWf+yxx8RTTz0lhBDiqaeeEo8//vhcS7Cr7373u+JHP/rRtMvi4+NFc3Ozkyu6th/84Afiv/7rv67Zxmw2i6SkJHHlyhUxMDAgMjIyxPnz551U4fT2798vhoaGhBBCPP744zO+70rZ5tfahqPddt955x2xfv16YbVaxZEjR0ReXp6cJQshhKirqxMnTpwQQgjR1dUlkpOTp7z3Bw4cELfffrsc5c1qtvdfidt8MrPZLCIjI0VlZeWEx5Wy3Q8ePChOnDgh0tPTxx6z5XP5Bj9XnNs9fcGCBUhNTZ3yeGFhIbZs2QJvb28kJibCaDSiqKho2nYPPPAAAOCBBx7Am2++ac/y5kQIgT/+8Y+45557ZKvBEcYPK6TVaseGBpJTfn4+NJrhfj3Lly9HbW2trPXMxpZtONPQTHKKiooaG3A3MDAQCxYsgMlkkrUme1LiNp/sgw8+wPz588cuJFeaVatWQaebOOu3LZ/Ljv5ccco5qpmGqZmssbERUVHDw3JERUWhqalpShtn+fjjjxEZGYnk5ORpl0uShPz8fGRnZ6OgoMDJ1c1sx44dyMjIwNatW6fdRbf1vZDLiy++iA0bNky7TCnb3JZtqPTtXFlZiZMnT2LZsmVTlh05cgRLlizBhg0bcP78eRmqm95s77/StzkA7N69e8Y/fpW63W35XHb4trdlt2v8DcD7AM5Nc/ubcW0+ApAz7v5vANw37v4LAL4yzXN3TLrfPtf67PgzPAfge9d4juiRr3oApwGsckStc6kdQCQANYb/APlPAC9Os/7XADw/7v7fAfi1nHWPa/MkgDcw0htVKdt8LtsQwL6Rr+8A+MK4Nh8AyJaj3mnqDwBwAsCd0ywLAhAw8v1GAGVy12vr+6/kbT5SjxZAC4BIJW93AAkAzo273zFpefs06zj0c2XO11EJIdbMdR0AtQBix903AKibpl2jJElRQoh6SZKiADhkl2q2n0GSJA2AOwFkX+M56ka+NkmS9AaAPACH7FnnDK9r0/aXJOl3AN6eZpGt74Vd2bDNHwBwB4DbxMhv+jTPIcs2n8aM21B8PmaZLNt5NpIkeQH4C4BXhRCvT14uhOga9/0eSZL+V5KkcCHEjQ/6eYNseP8Vuc3H2QCgRAjROHmBkrc7bPtcdui2d1b39LcAbJEkyVuSpEQAyQCmnqQabvfAyPcPAJDr5MkaABeFENOeLJEkyV+SpMDR7wHkY3jvQFYjv0SjNmP6mo4DSJYkKVGSJC2ALRje7rKRJGk9gH8GsEkI0TtDGyVtc1u24VsA7peGLQfQKYSQ9YSJNDxPwwsALgghfjFDm3kj7SBJUh6GPyNap2vrTDa+/4rb5pPcA2DXdAuUut1H2PK57NjPFTvvMm7GcLIOAGgEsH/csicBXAFwCcCGcY8/j5HDhADCMLy7XjbyVSfTru/vAWyf9Fg0gD0j3ydh+NDDaQDnATwpR53T1P0KgLMAzoz8kkRNrn3k/kYAl0feD9lrB/AZgBoAp0ZuO5W+zafbhgC2j/7eYPgixt+MLD+LcYfCZaz5Cxi+iP/MuG29cVLd3xnZvqcBHAWwUu66r/X+K32bj6vfD8PBEzzuMcVtdwwHaT2AoZHP8r+f6XPZmZ8rs41MQUREJCuOTEFERIrGoCIiIkVjUBERkaIxqIiISNEYVEREpGgMKiIiUjQGFRERKRqDioiInE6SpO2SJJ0auVVIknRgxra84JeIiOQyMv7khwB+LoT463RtuEdFRERy+iWAD2cKKQBzHz2diIjIHiRJ+jqAeAyPdThzOx76IyIiZ5MkKRvAHwDcLISYOsvrODz0R0REcvgOAB2AAyMdKp6fqSH3qIiISNG4R0VERIrGoCIiIkVjUBERkaIxqIiISNEYVEREpGgMKiIiUjQGFRERKRqDioiIFO3/A+SX1wH+CtU0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sympy.plotting.plot(lprime);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivative of the logistic function takes very small values at the long tails on each side of $z=0$. \n",
    "If our loss function has a $\\sigma^{\\prime}(z)$ factor (coming from the chain rule), this would lead to _slow learning_. \n",
    "We'd like to work with a better loss function, that avoids this problem, and we build one below by integration. (For a more detailed discussion, we recommend Chapter 3 of Michael Nielsen's free ebook [2]).\n",
    "\n",
    "It's important to note also that our prediction model is a nonlinear function, composed with the linear model, and the square-error would lead to a non-convex loss function that can have local minima, and make gradient descent fail. \n",
    "Here's an example posted on Stackoverflow in answer to this very [question](https://math.stackexchange.com/questions/2381724/logistic-regression-when-can-the-cost-function-be-non-convex). Consider just three data points, and a model with no intercept, $z = wx$: $(-1, 2), (-20, -1), (-5, 5)$. What would a square-error loss function look like? We can plot it using SymPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left(-5 + \\frac{1}{1 + e^{- 5 z}}\\right)^{2} + \\left(-2 + \\frac{1}{1 + e^{- z}}\\right)^{2} + \\left(1 + \\frac{1}{1 + e^{- 20 z}}\\right)^{2}$"
      ],
      "text/plain": [
       "(-5 + 1/(1 + exp(-5*z)))**2 + (-2 + 1/(1 + exp(-z)))**2 + (1 + 1/(1 + exp(-20*z)))**2"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "badloss = (1/(1+ sympy.exp(-z))-2)**2 + \\\n",
    "          (1/(1+ sympy.exp(-20*z))+1)**2  + \\\n",
    "          (1/(1+ sympy.exp(-5*z))-5)**2\n",
    "badloss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEgCAYAAADhUed1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsIklEQVR4nO3dd3wUdcIG8Gc2vfeySQiBEGBJISQLARQQTtqJiKCAAqKgKN6pJ/da7vW8O++163lweoI5iojtlENQQUSKKCWEhN5LerLpvW+yv/ePQIQjlexmZnef7+eTj7i7M/NkPgPPzuxvfyMJIUBERKQ0KrkDEBERtYUFRUREisSCIiIiRWJBERGRIrGgiIhIkVhQRESkSCwoIiJSJBYUEREpEguKiIgUyVbuAESmIEmSI4BpAMYACAJQB+AUgK1CiNNyZiOirpE41RFZGkmS/gLgTgA/AkgFUAjAEcBAAOOv/Pn3QogTMkUkoi5gQZHFkSTpDiHE1g6e9wcQKoRI6cVYRNRNLCiyWJIkjQFwQAjRfM1jcUKIIzLGIqIu4iAJsmTfA9gtSVLANY+tlisMEXWPqQtK8Ic/cv3ExsY6ffPNN2MHDx6cf+DAgauPDeuNbU+ZMkX2358//JHxxyg4io8sliRJmDZtGgYNGoQ5c+Zg0aJFkCSpV7ZdXFzcK9shsmS8xEcW6+rnqxEREdi3bx9++uknnDjBgXtE5sLUgyRMunKi7srKykJoaKjJt6PVapGSwkGCZLWMcqmCZ1BkcV5++WWUlpa2+VxoaCh2796Nb7/9tpdTEVF38TMosjjR0dG488474ejoiLi4OPj5+aG+vh4XL17EsWPHcPvtt+N///d/5Y5JRJ1gQZHF2bhxI/bv348333wT/v7+0Ol0cHd3x/z585GYmAgnJye5IxJRF7CgyOKkpqYiMzMTn3zyCfbs2XPdc3V1dSwoIjPBgiKL89hjj2HKlClIS0uDVqttfVwIAUmSkJaWJmM6IuoqjuIji7V06VKsXLmyy6+vr6/H2LFj0dDQgKamJtxzzz146aWXUFpaijlz5iAjIwNhYWH44osv4OXl1eG6OIqPrJxRRvGxoIiuEEKgpqYGrq6u0Ov1uPXWW7FixQps2rQJ3t7eeP755/H666+jrKwMb7zxRofrYkGRleMwcyJjkiQJrq6uAAC9Xg+9Xg9JkrBlyxYsXLgQALBw4UJs3rxZxpRE1oMFRXSN5uZmxMbGwt/fHxMnTkRCQgIKCgqgVqsBAGq1GoWFhW0um5iYCK1WC61Wi6Kiot6MTWSRWFBE17CxscGxY8eQk5OD5ORknDp1qsvLLlmyBCkpKUhJSYGfn58JUxJZBxYUURs8PT1x2223Yfv27QgICIBOpwMA6HQ6+Pv7y5yOyDqwoIiuKCoqQnl5OYCW70vt3LkTgwcPxvTp07F+/XoAwPr163HXXXfJmJLIevB7UERX6HQ6LFy4EM3NzTAYDJg9ezamTZuGUaNGYfbs2VizZg1CQ0Px5Zdfyh2VyCpwmDmRCXCYOVk5DjMnIiLLxYIiIiJFYkEREZEisaCIiEiRWFBERKRILCgiIlIkFhQRESkSC4qIiBSJBUVERIrEgiIiIkViQRERkSKxoIiISJFYUEREpEgsKCIiUiQWFBERKRILioiIFIkFRUREisSCIiIiRWJBERGRIrGgiIhIkVhQRESkSCwoIiJSJFtTrvyW13ejn68L+vm6IMzXBf2v/DfEywl2NuxGIiJqn0kLShvmhYziGmw+louq+qZfNqqS0MfbGf18XRAV7I4QL2cMCnBDRIArnO1NGomIiMyESdtgxdxhAAAhBEprGpFRUoO0ohpklNQgvbgG6UU1OJxRel159fF2wkB/NwwMdGstrXA/Vzja2ZgyKhERKUyvnK5IkgQfVwf4uDogvq/3dc81GwSySmtxoaAKF/KrcL6gChcLqrH3QhGaDAIAoJKAMB8XDAxww8AA19byCvNxgZ0tLxUSEVkiSQhhyvXf9Mr1zQZkFNfg/JXiulBQjQsFVcgoqcGV3oLawxH+bg6IDPZAVJAHooLdMSjQDQ62PNsieWm1WqSkpMgdg0guklFWotSCak+9vhmXi6pxTleJc/lVOJ1XiVO5Fai8cpnQViVhYIAbooLdMSLMG4PVLaXFQRnUm1hQZOWMUlBmNyLB0c4GkUEeiAzyaH1MCIHs0jqcyqvAqdwKnMytwM6zhTiUXorMklo42qkQE+KJYaGeGNbHC3GhnvB3d5TxtyAios6Y3RlUlzcsBHLKanEsuwJHsspwNKscp/MqoG9uiRTs6YTJkQHo5+uChP4+iPB3hSQZpfSJeAZF1s46L/H1RL2+GWd0lTiaVY6jWWUtAzMKqgEA3i72GBHmjYT+3kjo54PBgW5QqVhYdHNYUGTlWFA9JUTLCMJDaaVISi/BobRS5JbXAQA8nOxwZ4wamiB3jI3wQx9vZ5nTkjlhQZGVY0GZQk5ZLZLTS5GUVoLDGWVIL64BAPT3c8G4gX4YO9API/v5wMmeIwWpfSwosnIsKFMTQiCtuAZ7zxdh74UiJKWVoKHJAHtbFRL6eWNshC9u1wSgn5+r3FFJYVhQZOVYUL2tXt+M5PRS7L3QUlj6pmZkltZhYIArpkQGYkqUGhq1GwdbEAuKrB0LSm7ZpbXYdbYA20/nIzm9FAYBhHo7Y0pUICZHBmJYH08OtLBSLCiyciwoJSmpbsDOswXYfiof+y4VQ98sMCTIHQn9vDErLgSRQe48s7IiLCiyciwopaqs12PPuUKkZpbh8+RsNDYbMDDAFTPjQjAjNhiBHvySsKVjQZGVY0GZg/LaRnx7QodNR3JwJKsckgTcOsAXM+OCMTkykLcXsVAsKLJyLChzk15cg6+O5GDT0VzklNUh2NMJv9L4Y/Gt/dDXx0XueGRELCiyciwoc2UwCBzOKMXOs4VYfyADTQYDpkar8ejY/ogJ8ZQ7HhkBC4qsHAvKEhRU1mPd/gx8kpSJqoYmjA73waPjwjE2wpeDKswYC4qsHAvKklTV6/FZchbW7EtHQWUDBge64bFx4bgjRs1bhZghFhRZORaUJWpsMmDLsVwk/pSGi4XVCPZ0wqJb+2Hu8D5wceCACnPBgiIrZ5SC4ltzhbG3VeFebR98/7uxWLNQi2BPJ/zft2cw+vXdePv78yiubpA7osXKzs7G+PHjodFoEBkZiRUrVgAAjh07hpEjRyI2NhZarRbJyckyJyWyDjyDMgOpmWVI/OkydpwpgJ2NCvfEh+CRMf3Rz5cj/4xJp9NBp9MhLi4OVVVViI+Px+bNm/G73/0OTz/9NKZOnYpt27bhzTffxI8//tjhungGRVbOOu+oa43i+3rhgwVaXC6qxuqf07AxJQefJWdhSmQgHh0Xjtg+nnJHtAhqtRpqtRoA4ObmBo1Gg9zcXEiShMrKSgBARUUFgoKC5IxJZDV4BmWGCqvq8eH+DGxIykRVfRPmjwzFA6PCMDDATe5oFiMjIwNjx47FqVOnkJubi8mTJ0MIAYPBgAMHDqBv374dLs8zKLJyHCRh7aobmvDpoUxsSMpEfkU9Hh0bjt9OGABHO96rqieqq6sxbtw4vPDCC5g5cyaefPJJjBs3DrNmzcIXX3yBxMRE7Ny584blEhMTkZiYCAAoKipCZmZmb0cnUgoWFLUorWnEy1vPYNORXIT5OOPVu6MxeoCv3LHMkl6vx7Rp0zB58mQsW7YMAODh4YHy8nJIkgQhBDw8PFov+bWHZ1Bk5TiKj1p4u9jjndmx+HhxAgSA+1cfwv98eRxlNY1yRzMrQggsXrwYGo2mtZwAICgoCHv37gUA7N69GxEREXJFJLIqPIOyMPX6Zvxj10Uk/pQGdyc7vDhNgxmxwZyVogv27duHMWPGIDo6GipVy3u3V199Fe7u7njqqafQ1NQER0dHvP/++4iPj+9wXTyDIivHS3zUvnP5lfjDppM4mlWOMRG+eHlGFCek7UUsKLJyvMRH7Rsc6I6Nj43G/90ViaNZ5Zj095+w8sfL0Dcb5I5GRNQlLCgLZqOSsGBUGHYuG4fbBvnhje3ncOe7+3A0q0zuaEREnWJBWYFAD0d8sECLDxbEo7xWj5krD+DPW06hql4vdzQionaxoKzI5MhA/LBsLBaOCsNHSZmY+M5P+P50vtyxiIjaxIKyMm6OdvjL9EhsWjoans52eHRDKh7dkIL8inq5oxERXYcFZaWGhXrhmyduxXNTBuPH80W4/Z292H5KJ3csIqJWLCgrZmejwtLbwrHj6bEYP9gfj318BK9uOwuDgd8OICL5cTZzQl8fF/x99lB4Odsh8ac0ZJXU4u9zYuFkzzn9iEg+PIMiAICtjQovTY/Ei9OG4Psz+ZibeBCFVfxciojkw4KiVpIkYfGt/fDB/HhcKKjG3f88gAsFVXLHIiIrxYKiG0yKDMS/Hx2JxmYDZr1/APsuFssdiYisEAuK2hQT4onNv7kFQZ5OeHBdMv59OEvuSERkZVhQ1K5gTydsXDoKo8J98Nx/TuKN7ec4wo+Ieg0Lijrk5miHtQ8Ox30jQrHyx8t44rOjqNc3yx2LiKwAh5lTp+xsVHj17ij083XGq9vOIa+iDv96QAtfVwe5oxGRBeMZFHWJJElYMjYcK+fF4UxeJe5+fz8uFXKEHxGZDguKumVqtBqfLxmJusZmzHz/AA5c5gg/IjINFhR127BQL3z1+C0IcHfEA2uSsTE1R+5IRGSBWFB0U/p4O2Pj0tFI6O+N//nyON7ZcR5CcIQfERkPC4pumoeTHT58aATmaPvgH7sv4Xf/PsYRfkRkNBzFRz1iZ6PC67OiEerjjLe+P4+88jp8sEALbxd7uaMRkZnjGRT1mCRJ+M34AXj3vmE4nlOBReuSkVNWK3csIjJzLCgymjuHBuHTh0egsVnggTXJKK1plDsSEZkxFhQZlTbMBy/dFYnc8jo8tC4ZNQ1NckciIjPFgiKjGx7mjffuj8PJ3Ao89nEqGpsMckciIjPEgiKTmDgkAK/NjMbPF4vxP18e5ySzRNRtHMVHJjNneCiKqxvx1vfn4eNqjz9NGwJJkuSORURmggVFJvX4beEorm7Auv0Z8HNzwOO3DZA7EhGZCRYUmZQkSXjxjiEoqW7Em9vPw9fFAbOH95E7FhGZARYUmZxKJeHte4eirLYRz286AS8Xe0wcEiB3LCJSOA6SoF5hb6vCqvnxiA72wG8/PYLDGaVyRyIihWNBUa9xcbDF2geHI9jTCYs/PIxz+ZVyRyIiBWNBUa/ycXXA+kUj4GRvg4VrOSUSEbWPBUW9ro+3M9YvGoG6xmY8sJZTIhFR21hQJIvBge5Y8+Bw5JZxSiQiahsLimTDKZGIqCMsKJLVtVMiPbORUyIR0S/4PSiS3XVTIrk44MVpGk6JREQsKFKGq1Mird2fDl83e06JREQsKFKGG6ZEcnXAbC2nRCKyZiwoUoxrp0T6w6aT8Ha2x+2cEonIanGQBCnK1SmRooLc8RtOiURk1VhQpDicEomIABYUKRSnRCIiFhQpFqdEIrJuLChStN6cEik7Oxvjx4+HRqNBZGQkVqxY0frcu+++i0GDBiEyMhLPPvusyTIQ0S84io8U7+qUSI9uSMFjH6dizcLhsLc1/nsrW1tb/O1vf0NcXByqqqoQHx+PiRMnoqCgAFu2bMGJEyfg4OCAwsJCo2+biG7EMygyC70xJZJarUZcXBwAwM3NDRqNBrm5uVi5ciWef/55ODg4AAD8/f2Nvm0iuhELiszGnOGheGbyIGw5loeXt56FEKabty8jIwNHjx5FQkICLly4gJ9//hkJCQkYN24cDh8+3OYyiYmJ0Gq10Gq1KCoqMlk2ImvBS3xkVq6dEsnPzQFLbws3+jaqq6sxa9YsLF++HO7u7mhqakJZWRmSkpJw+PBhzJ49G2lpaTfMF7hkyRIsWbIEAKDVao2ei8jasKDIrFw7JdIb28/Bx9XeqFMi6fV6zJo1C/PmzcPMmTMBACEhIZg5cyYkScKIESOgUqlQXFwMPz8/o22XiG7ES3xkdq5OiTQmwhd/2HQSO88UGGW9QggsXrwYGo0Gy5Yta318xowZ2L17NwDgwoULaGxshK+vr1G2SUTtY0GRWTLFlEj79+/Hhg0bsHv3bsTGxiI2Nhbbtm3DokWLkJaWhqioKMydOxfr16/n7UCIeoFkyg+aAfDuc2RSJdUNuHfVQRRXN+DLx0ZjUKCb3JEAtHwGlZKSIncMIrkY5R0cz6DIrF07JdIDaw9xSiQiC8KCIrPHKZGILBMLiizCdVMifXjYpFMiEVHvYEGRxbg6JdLJnHIs/eQIGpsMckcioh5gQZFFuTol0k8Xikw2JRIR9Q5+UZcszpzhoSiubsRb35+Hj4sDXpym4bBwIjPEgiKL1BtTIhGRabGgyCKZekokIjI9k35RNzIyUjg5OZls/aZWVFRk1vOtMT8gBJBRUoPqhib09XGGu6OdkdJ17OTJk4iOju6VbZkCjx15mXv+1NTU00KIqJ6ux6RnUE5OTmb9bXpznw2A+VvUNDTh/n8l4Vx+FVY9nABtmLcR0nXMxcWF+15GzC8vSZLqjbEejuIji+fiYIu1Dw5HsKcTFn14GOfzq+SORERdwIIiq8ApkYjMj0kL6urN28wV88vL2Pl7c0okc78dB48deZl7fgCJxlgJZzMnq5OcXooFaw5hsNodnz6cABcH438Ua+6fIRD1EGczJ7oZI/pxSiQic9CjgpIk6V5Jkk5LkmSQJEnb3uu2b9+OQYMGYcCAAXj99ddbHy8tLcXEiRMRERGBiRMnoqysrCdxuq0r2z9//nzrzetiY2Ph7u6O5cuXAwD+8pe/IDg4+Lqb2yktPwCEhYUhOjoasbGx0Gq13V7eVLqy/ezsbIwfPx4ajQaRkZFYsWJF63M92f89mRKpveP5KiEEsrOzMWDAAMTExODIkSNdXrY3dJbhk08+QUxMDGJiYjB69GgcP3689bn2jqXe1Fn+H3/8ER4eHq3HxV//+tcuL9sbOsvw1ltvtWaPioqCjY0NSktbbsgp9/5ftGgR/P39ERXV9ghyIQSefPJJSJJ0SZKkE5IkxV19TpKkKZIknb/y3PNd2qAQ4qZ/AGgADALwIwBtG68RTU1Non///uLy5cuioaFBxMTEiNOnTwshhHjmmWfEa6+9JoQQ4rXXXhPPPvus6E3d3X5TU5MICAgQGRkZQggh/vznP4u33nrL5Dnb09X8ffv2FUVFRTe9vKl0Zft5eXkiNTVVCCFEZWWliIiIaD1+jLH/39t9UfR97lvx0tenhcFg6PT1HR3PV23dulW4u7sLg8EgDh48KEaMGNHlZU2tKxn2798vSktLhRBCbNu2rTW/EO0fS72lK/n37Nkj7rjjjpta1tS6m+Hrr78W48ePb/1/uff/3r17RWpqqoiMjGzz+a1bt4opU6YItFziGwngkGjpChsAlwH0B2AP4DiAIaKTjunRGZQQ4qwQ4nxHr0lOTsaAAQPQv39/2NvbY+7cudiyZQsAYMuWLVi4cCEAYOHChdi8eXNP4nRbd7e/a9cuhIeHo2/fvr2QrnM93X/msP/VajXi4lrehLm5uUGj0SA3N9doGR6/LRwP3RKGtfvTsWpvWqev7+h4vmrLli3w8fGBJEkYOXIkysvLodPpurSsqXUlw+jRo+Hl5QUAGDlyJHJycno1Y0d6sg/NZf9f67PPPsN9993Xiwk7NnbsWHh7t/89wi1btuCBBx7Alb5KAuApSZIawAgAl4QQaUKIRgCfA7irs+2Z/DOo3Nxc9OnzyxQzISEhrf/AFBQUQK1WA2j5h6iwsNDUca7T3e1//vnnNxws7733HmJiYrBo0aJev0TW1fySJGHSpEmIj49HYmJit5c3le5uPyMjA0ePHkVCQkLrYz3d/1enRJo+NAhvbD+HL1KyO3x9R8fzta+xt7e/4TVdWdbUupthzZo1mDp1auv/t3cs9Zau5j948CCGDh2KqVOn4vTp091a1pS6k6G2thbbt2/HrFmzWh+Te/935r9/PwA5AIKv/GS38XiHOh2+JEnSTgCBbTz1ghCi07cfoo1Rgr05s/Ttt9+O/Pz8Gx5/5ZVXurWexsZGfP3113jttddaH1u6dClefPHFln/kXnwRv//977F27doeZ76WMfLv378fQUFBKCwsxMSJEzF48GCMHTvWmDHbZaz9X11djVmzZmH58uVwd3cHYLz9r1JJePveoSirbcQfNp2Et7M9bh8S0OZru3I8t/cauf8uAN37+7hnzx6sWbMG+/bta31MzmMJ6Fr+uLg4ZGZmwtXVFdu2bcOMGTNw8eJFs9v/33zzDW655Zbrzljk3v+daev3Q8to7rZ+yU4/+O20oIQQt3ceq30hISHIzv6lOHNychAUFAQACAgIgE6ng1qthk6ng7+/f0821aadO3e2+1x3tv/dd98hLi4OAQEB1y1/1SOPPIJp06YZJ/Q1jJH/6v729/fH3XffjeTkZIwdO9Zs9r9er8esWbMwb948zJw587rlr+rp/re3VWHV/Hjc/68k/ObTI/iknSmROjqer33N2bNnb3hNY2Njp8uaWlfyA8CJEyfw8MMP47vvvoOPj0/r4+0dS72lK/mvvoEBgF//+td4/PHHUVxc3OXf3ZS6k6GtKzZy7//O/PfvByAEQB5aPnfq08bjHTL5Jb7hw4fj4sWLSE9PR2NjIz7//HNMnz4dADB9+nSsX78eALB+/XrcdVenlySNqjvbb+tasE6na/3zV1991e7IFlPpSv6amhpUVVW1/nnHjh2tOc1h/wshsHjxYmg0Gixbtuy654y9/7syJVJHx/O1v1dJSQmEEEhKSoKHhwfUanWXljW1rmTIysrCzJkzsWHDBgwcOLD18Y6OJSXlz8/Pb30nn5ycDIPBAB8fH7PZ/wBQUVGBvXv3Xvd3Qgn7vzPTp0/HRx99BKnFSAAVQggdgMMAIiRJ6idJkj2AuQC+7nSFnY2i6OgHwN1ouZbYAKAAwPdXHg8CsO3akR0RERGif//+4uWXX24d8VFcXCwmTJggBgwYICZMmCBKSkq6PaqkJ9rbfm5urpg6dWrr62pqaoS3t7coLy+/bvn58+eLqKgoER0dLe68806Rl5enuPyXL18WMTExIiYmRgwZMsTs9v/PP/8sAIjo6GgxdOhQMXToULF161YhhOn2f1ZJjRjxyg9ixCs/iOzSmhueb+t4XrlypVi5cqUQQgiDwSD8/PxE//79RVRUlDh8+HC7y9Y2NInj2aVi+6k88dWRbLH5SI7YdSZfnMurEE3NnY8qvBmd5V+8eLHw9PRs3d/x8fFCiI6Ppd7UWf53331XDBkyRMTExIiEhASxf//+DpdVWn4hhFi3bp2YM2fOdcspYf/PnTtXBAYGCltbWxEcHCxWr159w7H/+OOPC7SM2DuJa0Z3A/g1gAtXnntBdKFjOJMEURvO5Vdi9qqD8HVzwMbHRsPbxb7zha7R0UwS5bWN2HWuECkZpdCV1QEqCY36ZqhUEmxVEpoMAiqVhOr6JkQFe2DSkECMDveBSsW7ApPZMMrByoIiakdPpkRqq6DSiqqxZl86Dl4uQXltIyKDPDC8nzcGB7qhr7czPJzsYJAESqoakVFSjcMZ5bhUVI1GvQFujra4b0QoJg4J4O3ryRywoIhM7YczBXh0QwpujfDD6ge0sLft2se21xbUkcxSrNqbhh/OFsBOpcLMuGDMHxmKqGDPTtfT0NSMrSd0+OZ4HvacL8KtA3zxfzOi0M/XpSe/FpGpsaCIesO/D2fhuf+cxIzYILwzO7ZLl9q0Wi1e/2grVu29jIKqelTXN2H+yL5YMKov/N0cu51B32zAJ0mZWLMvHU0GgeenDsZdsZ1+jYRILkYpKJPeUZfIEswZHori6ka89f15+Lg64I93aNq9zFZc3YCvj+Uhs6QWD3+UgmBPJzwyph/u0YbA1eHmbzdvZ6PCg7f0w9QoNZ747Cie+vwYTuRU4A9TB8PWhnM+k2ViQRF1weO3haO4ugFr9qWjsk6Pkf294OvmCAigql4PXUU9DqaV4mhWGWobmyEgsHxOLO6IUcPOiAUS4OGITx5JwCtbz2LNvnSczqvAe/fHwdfVwWjbIFIKXuIj6iKDQeDLlGzsOleAxmYDAAkQLSPuIFq+RxUZ5IEJGn/cf8d4k98PatORHPxh00n4uNhj1YJ4xIR4mnR7RN3Az6CI5NDcbEBeRR3KahthMAi4Odkj2MMRjva/XJDorRsWnsqtwKMbUlFYVY/E+VqM1xh/NhCim8AbFhLJwcZGhT7eLogJ8UJsqDfC/VyvK6feFBXsgW+euBXTYoKw+KPD+OZ4p7PHEJnMqlWrEBsbC0mSjkmSlC5J0p6erI9nUEQm0Nu3fK9tbMKDaw8jNasMK+fFYVJkW/M7E/WOK9MZ7QbwphDim5tdD8+giCyAs70t1j40HDEhHvjtp0fx4/nevXUK0X9ZAWB3T8oJYEERWQxXB1t8+NAIDAx0xaMbUnHgUrHckcgKffjhhwDQF8BLPV0XC4rIgng42WHDogSE+bhg8foUHM4olTsSWZHU1FS8/fbbADBfCGHo6fpYUEQWxsvFHh8/nAC1pyMeWncYx7LL5Y5EVuK9995DaWkpAOy5MlBidU/Wx0ESRCbQ24Mk2pJfUY/ZHxxEeW0jPlsyEpFBHrLmIavCYeZE1L5AD0d8+kgC3BztsGBNMi4U3HgDRiIlY0ERWbAQL2d88nACbFUS5q0+hLSiarkjEXUZC4rIwoX5uuDTRxJgMAjMW30I2aW1ckci6hIWFJEVGODvho8fTkCdvhn3/SsJeeV1ckci6hQLishKaNTu2LAoARW1esxbfQiFlfVyRyLqEAuKyIpEh3jgw0UjUFBZj3mrD6GkukHuSETtYkERWZn4vl5Y++BwZJfV4k9bTqOoiiVFysSCIroiOzsb48ePh0ajQWRkJFasWHHd82+//TYkSUJxsflPITSyvw8+eigBu88V4t5VBzhwghSJBUV0ha2tLf72t7/h7NmzSEpKwj//+U+cOXMGQEt5/fDDDwgNDZU5pfGM6O+Njx8egdKaRty76iAu8ntSpDAsKKIr1Go14uLiAABubm7QaDTIzc0FADz99NN48803IUlG+YK8YsT39ca/Hx2FJoPA7A8O4jinRSIFYUERtSEjIwNHjx5FQkICvv76awQHB2Po0KEdLpOYmAitVgutVouioqJeStpzGrU7Nj42Ci4Otrj/X0k4cNn8L2GSZeBcfET/pbq6GuPGjcMLL7yAKVOmYPz48dixYwc8PDwQFhaGlJQU+Pr6drgOJczF1135FfVYsOYQMktr8d59w3jTQ+oJzsVHZGx6vR6zZs3CvHnzMHPmTFy+fBnp6ekYOnQowsLCkJOTg7i4OOTn58sd1egCPRzxxaOjoFG7Y+knR/Cf1By5I5GV4xkU0RVCCCxcuBDe3t5Yvnx5m6+x5DOoq6obmrDkoxQcuFyCP00bgkW39pM7EpkfnkERGdP+/fuxYcMG7N69G7GxsYiNjcW2bdvkjtXrXB1ssfbB4Zg0JAB//fYM/v7DBZj4jSxRm3gGRWQC5nwGdVVTswHPbzqJjak5eHB0GP40bQhUKssaxUgmY5QDxdYYKyEiy2Nro8Kbs2Lg4WSHNfvSUVmnxxv3xMDOhhdeqHewoIioXSqVhD/eoYGXsx3e3nEBlfVNeO/+YXC0s5E7GlkBvhUiog5JkoTfTojAX++KxM6zBXhwXTKq6vVyxyIrwIIioi55YFQYls+JxeGMMsxbfQilNY1yRyILx4Iioi6bMSwYiQvicT6/CveuOsAbH5JJsaCIqFt+pQnAR4tGoLCyAfeuOoi0omq5I5GFYkERUbcl9PfBZ0tGol7fjNkfHMTpvAq5I5EFYkER0U2JCvbAF4+Ngr2NCnM/SMLhjFK5I5GFYUER0U0L93PFl0tHw8/NAQvWHMKec4VyRyILwoIioh4J9nTCF4+NQrifKx75KAVfH8+TOxJZCBYUEfWYr6sDPlsyEnF9vfDU50fxcVKm3JHIArCgiMgo3B3t8NGiEZgwyB9/3HwK/9xziZPMUo+woIjIaBztbLBqQTxmxAbhre/P4/XvzrGk6KZxLj4iMio7GxXemR0Ldyc7fPBTGspr9Xh1ZjRsOBM6dRMLioiMTqWS8NL0SHg62eEfuy+hsl6P5XNj4WDLSWap63iJj4hMQpIkLJs0CH+8Q4PvTuXj4fUpqGlokjsWmREWFBGZ1MNj+uOte2Kw/1Ix5q85hPJaTjJLXcOCIiKTu1fbByvnx+N0biXmfJCEwsp6uSORGWBBEVGvmBwZiHUPDUd2WS3uWXUQWSW1ckcihWNBEVGvuWWALz59ZCQq6/W4Z9UBnM+vkjsSKRgLioh6VWwfT3zx6ChIEvDHzSfxn9QcfleK2sSCIqJeNzDADZuWjoazvS1+/+VxzFt9iPeVohuwoIhIFsFezlj34HC8cncUTuZWYMqKn/GPXRfR0NQsdzRSCBYUEclGpZIwL6Evdv1+HCZHBuKdHy7g1yt+xqG0ErmjkQKwoIhIdv5ujnj3vmH48KHhaGw2YE5iEp7deBxlNfzOlDVjQRGRYtw2yB87fjcOS28Lx6YjufjVO3ux6QgHUVgrFhQRKYqTvQ2emzIY3z55K8J8nLHsCw6isFYsKCJSpMGB7tj42GgOorBiLCgiUiwOorBuLCgiUjwOorBOLCgiMhscRGFdWFBEZFbaGkSx9JNUnMgplzsaGZlk4ncefFtDVkmr1SIlJUXuGBbPYBD47HAWtp3UYf+lEowb6IcnJgyANsxb7mjWTjLKSlhQRMbHgupdVfV6bEjKxOqf01Fa04hR/X3wxIQBGBXuA0kyyr+V1D0sKCKlYkHJo7axCZ8eykLiT2korGpAfF8vPDFhAMYN9GNR9S4WFJFSsaDkVa9vxpcp2Vj542XkVdQjJsQDvx0fjl9pAmGjYlH1AhYUkVKxoJShscmAr47m4P0fL8PR1gbVDU24PyEUc4b3ga+rg9zxLBkLikipWFDK0tRswM6zBdiQlIn9l0pgb6PCr6MDsWBUX8SFevHyn/GxoIiUigWlXJcKq/FxUib+k5qDqoYmaNTuWDCyL2YMC4Kzva3c8SwFC4pIqVhQylfb2IQtx/Lw0cFMnNVVYnS4D0K8nDBjWDBG9vOBip9V9QQLikipWFDmQwiBI1nl+O6kDp8lZ6GmsRlqD0dMjw3C3cOCMTjQXe6I5ogFRaRULCjzVNfYjJ1nC7D5aC72XihCk0FgcKAbZgwLxh0xavTxcpY7orlgQREpFQvK/JXWNGLriTx8dTQXJ3Mq4Ghvg74+zpg0JBCTIgMwKMCNgyvax4IiUioWlGXJKqnFtlM67DidjyNZ5QBwpawCMCkyEMP6eMLWhlObXoMFRWRM2dnZeOCBB5Cfnw+VSoUlS5bgqaeewjPPPINvvvkG9vb2CA8Px7p16+Dp6dnhulhQlquwsh4/nC3AjtMFOHC5GE3NAn28nREd4oGxEb4YE+GHIE8nuWPKjQVFZEw6nQ46nQ5xcXGoqqpCfHw8Nm/ejJycHEyYMAG2trZ47rnnAABvvPFGh+tiQVmHqno99l8qxg9nCvHzxSIUVjUAAML9XDAmwg9jB/oioZ8PXBysbvi6UQrK6vYaUXvUajXUajUAwM3NDRqNBrm5uZg0aVLra0aOHImNGzfKFZEUxs3RDlOi1JgSpYYQAhcKqvHzxSL8dLEYnyVn4cMDGbCzkXBXbDCCPJ0wIswbw0I9rbGwbgr3ElEbMjIycPToUSQkJFz3+Nq1azFnzhyZUpGSSZKEQYFuGBTohofH9Ee9vhkpGWX4+WIRcsrr8N7uizAIwEYlISrIHcPDvKEN88bwMC/4cNqlNvESH9F/qa6uxrhx4/DCCy9g5syZrY+/8sorSElJwaZNm9ocvZWYmIjExEQAQFFRETIzM3stMylfVb0eR7LKcTi9FMkZpTiWXY7GJgN8Xe3h6mCL2D6eGHrlZ4jaHY52NnJH7gl+BkVkbHq9HtOmTcPkyZOxbNmy1sfXr1+PVatWYdeuXXB27vy7MPwMijrT0NSMkzkVOKOrxL6LxTiWXd76GZadjQSN2h2jw33Qz9cFQ9QeiAhwNafSYkERGZMQAgsXLoS3tzeWL1/e+vj27duxbNky7N27F35+fl1aFwuKbkZ+RT2OZZfjeE45jmeXQyUB+y6VAABsVRIG+LtiiNodQ4LcoQl0w2C1u1IvD7KgiIxp3759GDNmDKKjo6FStXyn5dVXX8WTTz6JhoYG+Pj4AGgZKLFq1aoO18WCImMwGASyy2pxOq8SZ/IqcUZXidN5FSiobIC3sx1Ka/XwdbXHAH9XDAxwQ0SAGwb6uyIiwA3eLvZyRmdBESkVC4pMqbi6ARfyq3BGV4kLBVW4UFCNS4XVqG5oan2Nr6s9Jgz2h52NCv18XdDfzwX9fF0R4uUEO9N/qZjDzImIrJGvqwN8Bzhg9ADf1seEENBV1ONCQRUuFlTjYmEV9M0CO87oUF6rb32drUpCqLczRoX7wN5WhT5ezgj1dkYfb2f08XZS1C1HlJOEiIhumiRJCPJ0QpCnE24b5H/dc2U1jUgrrkF6cQ3SiqqRXlyDyjo9dp0rRG1j83Wv9XW1R0hraTm1/NerpcDUHo69OqUTL/ERmQAv8ZE5EEKgtKYRWaW1yC6rQ3ZpLbJLa6/8fy3yyuvRbPjln/GEft7ILq1FoIcj1B5OV/57/Z/93Rxga2Ocm2nxDIqIyEpJkgQfVwf4uDpgWKjXDc83NRugq6hvLa3yOj0uFlQjv7IOZ/MrsftcIer015+BqSQg7bU7jJKPBUVERG2ytVFd+WzKGaPbeF4Igcq6Jugq66CrqEd+RT105XXG277R1kRERFZFkiR4ONvBw9nOJHce5g1MiIhIkVhQRESkSCwoIiJSJBYUEREpEguKiIgUiQVFRESKxIIiIiJFYkEREZEisaCIiEiRWFBERKRILCgiIlIkFhQRESkSC4qIiBSJBUVERIrEgiIiIkViQRERkSKxoIiISJFYUEREpEgsKCIiUiQWFBERKRILioiIFIkFRUREisSCIiIiRWJBERGRIrGgiIhIkVhQRESkSCwoIiJSJBYUEREpkiSEkDsDkcWRJGm7EGKK3DmIzBkLioiIFImX+IiISJFYUEREpEgsKCIiUiQWFBERKRILioiIFOn/AadUcGBKwfTdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sympy.plotting.plot(badloss, xlim=(-1,1));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see a local minimum, where gradient descent could get stuck!\n",
    "\n",
    "But let's go back to inspecting the derivative of the logistic function. \n",
    "We note that $\\sigma^{\\prime}(z)$ has the same expression in the denominator as $\\sigma(z)$, but squared. Let's play around with this..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{e^{- z}}{1 + e^{- z}}$"
      ],
      "text/plain": [
       "exp(-z)/(1 + exp(-z))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lprime/logistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, we can try to express this in terms of $\\sigma(z)$: add and subtract $1$ to the numerator and factor:\n",
    "\n",
    "$$\\frac{e^{-z}}{1+e^{-z}} = \\frac{1+e^{-z}-1}{1+e^{-z}} = 1 - \\frac{1}{1+ e^{-z}} = 1 - \\sigma(z)$$\n",
    "\n",
    "We get this interesting property for the derivative of the logistic function:\n",
    "\n",
    "$$\\sigma^{\\prime}(z) = \\sigma(z) (1-\\sigma(z))$$\n",
    "\n",
    "Following Nielsen [2], let's call $\\sigma(z)=a$, representing the predicted value by the model where $z=wx +b$. \n",
    "The deviation between the model output and the observational data at each point is measured by $|y-a|$. \n",
    "A square-error loss function, as used for linear regression, is $(y-a)^2$, giving derivatives with respect to the parameters that are proportional to the error: \n",
    "the larger the error, the larger the step taken in gradient descent. \n",
    "That's a nice property.\n",
    "\n",
    "Seeking a loss function $L$ with the same behavior, we would like, for example:\n",
    "$\\frac{\\partial L}{\\partial b} = (y-a)$. \n",
    "Using the chain rule,\n",
    "$$\\frac{\\partial L}{\\partial b} = \\frac{\\partial L}{\\partial a}\\frac{\\partial a}{\\partial z} = \\frac{\\partial L}{\\partial a} \\sigma^{\\prime}(z) = \\frac{\\partial L}{\\partial a} \\, a(1-a)$$\n",
    "\n",
    "using the property of $\\sigma^{\\prime}$ in the last step.\n",
    "\n",
    "Our desired behavior for $L$ is that this derivative be $(y-a)$, giving:\n",
    "\n",
    "$$ \\frac{\\partial L}{\\partial a} = \\frac{(y-a)}{a(1-a)}$$\n",
    "\n",
    "\n",
    "This lets us construct $L$ by integration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "a, y = sympy.symbols('a y', real=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{- a + y}{a \\left(1 - a\\right)}$"
      ],
      "text/plain": [
       "(-a + y)/(a*(1 - a))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dLda = (y-a)/a/(1-a)\n",
    "dLda"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle y \\log{\\left(a \\right)} + \\left(1 - y\\right) \\log{\\left(a + \\frac{1 - 2 y}{2 y - 1} \\right)}$"
      ],
      "text/plain": [
       "y*log(a) + (1 - y)*log(a + (1 - 2*y)/(2*y - 1))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L = sympy.integrate(dLda, a)\n",
    "L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle y \\log{\\left(a \\right)} - \\left(y - 1\\right) \\log{\\left(a - 1 \\right)}$"
      ],
      "text/plain": [
       "y*log(a) - (y - 1)*log(a - 1)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sympy.simplify(L)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Plus a constant of integration.) We note that since $a<1$ (and the logarithm is only defined for positive real numbers) we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle - y \\log{\\left(a \\right)} + \\left(y - 1\\right) \\log{\\left(1 - a \\right)}$"
      ],
      "text/plain": [
       "-y*log(a) + (y - 1)*log(1 - a)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L = -y*sympy.log(a) + (y-1)*sympy.log(1-a)\n",
    "L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{- a + y}{a \\left(a - 1\\right)}$"
      ],
      "text/plain": [
       "(-a + y)/(a*(a - 1))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sympy.simplify(L.diff(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivative is the same expression we started with. \n",
    "This is the _logistic loss function_ for a single data point:\n",
    "\n",
    "$$L = -y \\log(a) + (y-1) \\log(1-a)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect this loss function. If the true value is $y=1$, we get $L= -\\log(a)$. \n",
    "Plotting in the range of interest shows that the gradient is largest when the model output is furthest away from the true value. \n",
    "This is a nice property."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEhCAYAAAAqDTTQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAidUlEQVR4nO3deXTc5X3v8fczo320b7ZleZEsDN5kg4UtQ1jTJA44NzRJW5L0hIRyXSdNG3rak6TNbXNJm57Qe27arFBugZsF2tukl0trbBM2Q3AwRga8G+Pd8qZ93zXf+8eM5AVjjW1Jv9/MfF7n6MxI81u+eg7Wh+f3PL/n58wMERERvwl4XYCIiMiFKKBERMSXFFAiIuJLCigREfElBZSIiPiSAkpERHxJASUiIr6kgBIREV9SQImIiC+ljLWBcy4DWAXcBJQBvcBO4Bkz2zWx5YmISLJyF1vqyDn334GPARuBrUADkAHMBW6Lvv8zM9s+0YWKiEhyGSug7jSzZy7yeSkw08zqJqI4ERFJXhcNKBEREa+MOQYF4JwrAb4GzCdyWQ8AM7t9guoSEZEkF+ssvieAPUAF8ABwGHhjgmoSERGJ7RKfc26rmS11zm03s+roz142s1smvMKzrFy50jZs2DCZpxQRkSvnLmenmC7xAYPR15POuTuBE0D55ZzwSjQ1NU32KUVExCOxBtTfOufygD8DfgDkAn86YVWJiEjSiymgzGxt9G07kfufREREJtRFJ0k45/6bc67wIp/f7pxbdbknd87lO+d+6Zzb65zb45xbcbnHEhGRxDJWD2oH8J/OuT7gTaCRyDTzq4AlwPPA313B+b8HbDCzTznn0oCsKziWiIgkkLEC6lNmdqNz7qtEljmaBnQAPwdWm1nv5Z7YOZcL3Ax8HsDMBoCByz2eiIgklrECaqlzbhbwWd479pRJZOHYy1VJpEf2uHNuMZG1/r5iZt1XcEwREUkQY92o+zCwAbgGqDvra2v09UqkANcBD5nZtUA38PXzN3LOrXbO1Tnn6hobG6/wlCIiEi9ivVH3ITP74rie2LmpwGYzmx39/ibg62Z25/vtU1NTY3V1WpdWRCTOXNaNujEtdTTe4RQ95ingmHPu6uiPPgjsHu/ziIhIfIr1Rt2J8sfAE9EZfAeBL4y1Q//QMOkpwQkvTEREvOXpI9/N7G0zqzGzajO7y8xaL7b9oaZuPvu/Xp+s8kRExEOeBtSlCjhHV/+Q12WIiMgkiK+ACqCAEhFJEnEVUEH1oEREkkZcBVTAOboVUCIiSSG+AirgGBw2+oeGvS5FREQmWFwFVDB6q1dXn3pRIiKJLq4CKhCIJFR3v3pQIiKJLq4CKugiAdXZPzjGliIiEu/iKqDUgxIRSR7xFVDRHlSXelAiIgkvrgIqGBgJKPWgREQSXVwFVECz+EREkkZ8BdToGJQCSkQk0cVVQJ2ZxaeAEhFJdHEVUABZaUH1oEREkkDcBVR2eooCSkQkCcRlQOkSn4hI4ou/gMpQD0pEJBnEXUCF0lI0zVxEJAnEXUBlZ6TooYUiIkkg/gIqXQElIpIMUrw8uXPuMNAJDANDZlYz1j7XTM3heFsvw2EbXfpIREQSjx96ULeZ2ZJYwgmgNDedLYda2HuqY6LrEhERD/khoC7J8ooiADYfbPG4EhERmUheB5QBv3LObXXOrY5lh7L8TGYUZvL6weYJLk1ERLzk6RgUcKOZnXDOlQLPOef2mtkrZ28QDa7VADNnzgSgtqKI5/acJhy20QVkRUQksXjagzKzE9HXBuApYNkFtnnEzGrMrKakpASA5ZVFtPUMsq+hc1LrFRGRyeNZQDnnQs65nJH3wIeBnbHsu7yiEIDXNQ4lIpKwvOxBTQFedc5tA7YAz5jZhlh2nFGYxfT8TDZrHEpEJGF5NgZlZgeBxZe7//KKQl7e14iZ4ZzGoUREEo3Xs/guW21lEc3dA+xv6PK6FBERmQBxG1DLKyPjUJsPaRxKRCQRxW1AzSzMYmpuhu6HEhFJUHEbUM45llcW8vqhFszM63JERGScxW1AQWTZo8bOfg42dXtdioiIjLO4DqjaSt0PJSKSqOI6oCqKQ5TkpPP6IY1DiYgkmrgOKOccyysKef2gxqFERBJNXAcURNblO9XRx9GWHq9LERGRcRT3AVUbXZdPyx6JiCSWuA+oqtJsikJpmighIpJg4j6gzr4fSkREEkfcBxRE7oc63tbLMY1DiYgkjMQIqJH7odSLEhFJGAkRUHNLc8jPStW6fCIiCSQhAioQcCybXchm3bArIpIwEiKgIPJ8qGMtvZxo6/W6FBERGQcJE1BnxqHUixIRSQQJE1DXTM0lNyNF90OJiCSIhAmoYMCxrEL3Q4mIJArPA8o5F3TOveWcW3ulx1peUcShpm5Od/SNR2kiIuIhzwMK+AqwZzwOVFtZBGhdPhGRROBpQDnnyoE7gX8ej+PNL8slJz1Fl/lERBKA1z2ofwS+CoTH42DBgKNmdoFu2BURSQCeBZRzbhXQYGZbx9hutXOuzjlX19jYOOZxl1cWcaCxm8bO/vEqVUREPOBlD+pG4L845w4D/wrc7pz7+fkbmdkjZlZjZjUlJSVjHnR59PlQW3SZT0QkrnkWUGb2F2ZWbmazgbuBF83s96/0uAun5xFKC2qihIhInPN6DGrcpQYDLJ1dqBUlRETinC8Cysw2mtmq8Tre8opC9p3uoqV7YLwOKSIik8wXATXeaitHxqHUixIRiVcJGVCLpueTkRpgs9blExGJWwkZUGkpAWpmFWqihIhIHEvIgILIONQ7pztp69E4lIhIPErcgKoswkz3Q4mIxKuEDajFM/JITwloXT4RkTiVsAGVnhLk2pn5uh9KRCROJWxAQeTxG7tPdNDeO+h1KSIicokSOqCWVxQRNqg7rMt8IiLxJqED6tqZ+aQFNQ4lIhKPEjqgMlKDLJmRr+dDiYjEoYQOKIDllYXsPNFBV/+Q16WIiMglSPiAqq0sYjhsGocSEYkzCR9Q180sIDXotC6fiEicSfiAykwLUl2u+6FEROJNwgcURNbl21HfTs+AxqFEROJFcgRUZRFDYWPrkVavSxERkRglRUDVzCogGHC8rnEoEZG4kRQBFUpPYdH0PI1DiYjEkaQIKIjcD/X2sTZ6B4a9LkVERGKQNAFVW1HE4LDx1lGNQ4mIxAPPAso5l+Gc2+Kc2+ac2+Wce2Aiz1czu4CAg81al09EJC6keHjufuB2M+tyzqUCrzrn1pvZ5ok4WU5GKgun57HlUDPhsBEIuIk4jYiIjBPPelAW0RX9NjX6ZRN5zj//8NVsPtjC/3zunYk8jYiIjANPx6Ccc0Hn3NtAA/Ccmb0+kee76apiPr1sJj966QBPvVU/kacSEZEr5GlAmdmwmS0ByoFlzrmF52/jnFvtnKtzztU1NjZe0fmcc3zr4wuorSzka7/coRt3RUR8zBez+MysDdgIrLzAZ4+YWY2Z1ZSUlFzxuVKDAR767FLK8jP4w5/VUd/ac8XHFBGR8eflLL4S51x+9H0m8FvA3sk4d0EojX++53r6h8Lc95M6PStKRMSHvOxBTQNecs5tB94gMga1drJOXlWazY8/ex3vNnTxlX95i+HwhM7PEBGRS+TlLL7tZnatmVWb2UIz+9Zk13DTVSV882PzeWFvA3+/YVI6byIiEiMv74Pyhc+tmM3+hi7+6ZWDzCnN5ndrZnhdkoiI4JNJEl7761Xz+UBVMd94agdbtNKEiIgvKKCAlGCAH33mOmYUZvGHP6vjaLNm9omIeE0BFZWXlcqj91xP2OAPfvIGHX2DXpckIpLUFFBnqSgO8dDvX8ehpm7++Mm3GBoOe12SiEjSUkCd54Y5xTzw8QW8vK+Rv1unmX0iIl5J+ll8F/LZ5bPY39DFY5sOUVWazWeWz/S6JBGRpKMe1Pv4xh3zuGVuCX/99E5+c6DJ63JERJKOAup9pAQD/OAz1zK7OMQXf/4mh5q6vS5JRCSpKKAuIjcjlUfvqSHgIjP72ns0s09EZLIooMYwqyjEw7+/lGMtPfzRk28yqJl9IiKTQgEVg+WVRXz7rkW8ur+Jv1m72+tyRESSgmbxxeh3r5/B/sYuHnnlIBXFIb5wY4XXJYmIJDQF1CX42sprONHWy7/VHeNoSw9fW3kNGalBr8sSEUlIusR3CYIBx3d/dwnLZhfy+KbDfOwHr7LzeLvXZYmIJCQF1CVKSwnwwMcX8pN7l9HeO8hv/3gTP964Xw88FBEZZwqoy3TL3BKevf9mPjx/Kn+/4R1+759e41iLVkEXERkvCqgrUBBK44efuZZ/+L3FvHOqk5X/+Ar/VncMM/WmRESulALqCjnn+O1ry9nwpzezqDyPr/5yO2t+vpXmrn6vSxMRiWsKqHEyPT+TJ++r5S/vuIaX9jbykX/8NS/uPe11WSIiccuzgHLOzXDOveSc2+Oc2+Wc+4pXtYyXQMCx+uY5PP3lGynOTuPe/13HN57aQc/AkNeliYjEHS97UEPAn5nZPKAW+CPn3HwP6xk386bl8vSXb2T1zZU8ueUod37/Vd462up1WSIiccWzgDKzk2b2ZvR9J7AHmO5VPeMtPSXIX94xjyfvq2VgKMynHn6Nf3hun9byExGJkS/GoJxzs4Frgdc9LmXcrZhTxPr7b+Lji8v43gvv8qmHfsPBxi6vyxIR8T3PA8o5lw38O3C/mXVc4PPVzrk651xdY2Pj5Bc4DnIzUvnu7y3hR5+5jiMtPdzx/V/zs81HNB1dROQinJd/JJ1zqcBa4Fkz++5Y29fU1FhdXd3EFzaBTnf08ee/2Mahpm7ys1L50q1VfGTBVIIB53VpIiIT5bL+wHkWUM45B/wEaDGz+2PZJxECCsDMeHbXKb6zfi+Hm3uoLAmx5pY53LVkOmkpnndqRUTGW9wF1AeAXwM7gJGZA39pZuveb59ECagRw2Fj/c6T/PilA+w+2cG0vAz+602V3L1sBllpWmheRBJGfAXU5Ui0gBphZry8r5EfbzzAlkMtFGSl8oUbK7hnxWzyslK9Lk9E5EopoBLB1iMt/PilA7ywt4FQWpDP1s7iDz5QwZTcDK9LExG5XAqoRLL3VAcPbTzAf247QUogwCeXlrPmlkpmFYW8Lk1E5FIpoBLR0eYe/umVA/xiaz1Dw2HurC7ji7fMYX5ZrteliYjESgGVyBo6+nh00yGe2HyUrv4hbru6hC/dVsX1swu9Lk1EZCwKqGTQ3jPIzzYf5rFNh2npHuD62QV86dYqbr26hMjMfRER31FAJZPegWH+zxtHeeSVg5xo72PetFy+eOsc7lw0TTf9iojfKKCS0eBwmKffPsFDG/dzoLGbWUVZ3PeBSj553XSy0nUvlYj4ggIqmYXDxq92n+ahjfsZDIc53NTDB+dN4c5F07j16hIyUoNelygiyUsBJZGbft882sb/fbOe9TtP0dI9QHZ6Cr81r5RV1WXcNLeY9BSFlYhMKgWUnGtoOMzmgy2s3X6CDbtO0dYzSE5GCh+eP5VV1dO4sapYa/+JyGRQQMn7GxwOs2l/E89sP8mzu07R0TdEXmYqH1kwhVXVZayYU0RqUGElIhNCASWxGRgK8+t3G3lm+0l+tfs0Xf1DFGSlsnLhNFZVT2N5RSEpCisRGT8KKLl0fYPDvLKvkbXbT/L8ntP0DAxTnJ3GyoVTuXNRGcsqCjVtXUSulAJKrkzf4DAv7W1g7Y6TvLingd7BYUpy0rlj4VRWLS5j6cwCAgorEbl0CigZPz0DQ7y4t4G1207y0jsN9A+FmZqbwR2LprFq8TSunZGvlStEJFYKKJkYXf1DvLDnNGu3n+TldxoZGA4zPT+TOxZN5Y5F01iisBKRi1NAycTr6Bvk+d2RsPr1u41UFIdo6R5geWURKyqLWDGniMrikAJLRM6mgJLJ1d4zyG8ONPHc7tP85kAzpzr6AJiSmz4aVisqi5lRmKnAEkluCijxjplxuLmH1w4089rBZl470ExTVz8A0/Mzo2EVCa2y/EyPqxWRSaaAEv8wM/Y3dI2G1WsHm2nrGQRgVlEWN8wpojYaWKU5epy9SIJTQIl/hcPG3lOdo4H1+qFmOvuGAJhTEuKGOcWsiIZWYSjN42pFZJzFX0A55x4DVgENZrZwrO0VUIljOGzsOtE+2rt641AL3QPDAFwzNYfayiJumFPE8ooi8rJSPa5WRK5QXAbUzUAX8FMFVHIbHA6zvb6dzdEeVt2RFvoGwzgHC8pyWVFZxA1zirm+opBsPedKJN7EX0ABOOdmA2sVUHK2/qFh3j7aNnpJ8K2jbQwMhwkGHIum5/HBeaVcPSWH6vJ8puZpDEvE5xRQkrj6BofZeqR19JJgRkqATQeaASjNSae6PI9F0/Mjr+V5FGene1yxiJwlMQPKObcaWA0wc+bMpUeOHJnE6sSvegeG2X2yne317eyob2f78XYONHYx8p9zWV4G1eX5LCrPi4ZXHvlZmnwh4pHEDKizqQclF9PZN8iuEx2jgbWjvo3DzT2jn88szGJReR6Lo72thdNzycnQBAyRSXBZAaXRZkkYORmp1FZGpqqPaO8ZZOeJdrbVt7Gjvp23j7bxzPaTo59XloSonp5HdXnk8uD8slyy0vTPQsQPvJ7F9y/ArUAxcBr4ppk9+n7bqwcl46G5q58dxyOXBrfVt7PjeBunOyKrXgQcXFWac6anVZ7PNVNzyEgNely1SFyLz0t8l0IBJRPldEffOZcGt9e309w9AEBKwPGh+VPITAsyd0oOV0/J4aop2UzP1xqDIjFSQImMFzPjZHsf26Nh1d47yPN7To/2tABCaUGqpuQwtzSbudHQunpqDlNzMxRcIudSQIlMtPaeQd5t6GTf6S72ne6MfnWNLowLkJOewlVTRkIrh7nR96U56QouSVYKKBGvtHYPRMKqoYt9pyLB9W5DFy3Ry4QAeZmpzJ2SHQmt0jMBVpydpuCSRKeAEvGbpq7+SHCdioTXu9EeV3vv4Og2haE0rooG1miATcnRormSSDTNXMRvirPTKc5O54Y5xaM/MzMaO/t5JxpW70YvFf6/t47T2T901r5p0dDKGb1kOKsoi5JsXSqU5KCAEplkzjlKczMozc3gpqtKRn9uZpzq6OOdU528OzLG1dDFL+qO0T0wzMLpuew83kEoLcisohAVxSFmF2edeV8U0uVCSSi6xCfic+GwcaK9l4ONXRxs7OZwcw+Hmro50tzNsdZehsNn/g1np6cwqyiL2cUhKopCzCrKigZZiKKQwks8ozEokWQzOBymvrWXw83dHG6KfjX3cLi5m/rzwisnPYVZxVnMjva4Ij2vyPeFCi+ZWBqDEkk2qcEAFcWRwOHqcz8bGApT39rDkWiP63BzJLy217ezbsdJzsoucjJSmF0Uiva8IpcNZxeHmFmYSbHGvMQj6kGJJKGR8Drc3M2hph6ONHePhtjx1t7R8Jqen0lL9wDlBZnRr6zzXjPV+5JYqAclIrFJSwlQWZJNZUn2ez4bGApzrLWHw02Ry4THWno41tpDfWsvbx5tO2eKPEBmanA0rGYUnh9gWRRkpSrA5LIooETkHGkpAeaUZDPnAuEF0NE3yPHWXupbe6mPBlf9RQIsKy34nl7XyPsZBVnkK8DkfSigROSS5GakkjstlXnTci/4eXvvSICNhNeZ93WHW+joGzpn+1Ba8D3hNaMwk2l5mUzLy6A4O51AQAGWjBRQIjKu8jJTyctMZX7Z2AF27Lwg23Kohc7+Ia4qDfFuQzcAqUHHlNwMpuVljIbWtLwMpuZlUpafwdS8DIpDCrFEpIASkUkVW4D1cKKtj5PtvZxs74t+9bKtvo0Nu/oYGAqfs89IiJXlZTItGlpleZnnvBaF0hRicUYBJSK+EgmwPOaX5V3wczOjpXvgnOA62d7HybbI61tH2zjV3sfA8LkhlhYMMCUv/axe2Jne2LS8TEpz0ykKpZESDEzGrykxUECJSFxxzlGUnU5RdjoLp79/iDV3D3CqvY8Tbb2c6ujjRFsfp9p7OdHex5tHWznVfpLB4ch8+ppZBdQdaSXgoDCUTmlOOqW50decDEpyzv5Z5Hs9ZXniKaBEJOE450YX6n2/EAuHz4RYQ2cfJ9r7aOzoo6Gzn8bOfho6+9lzsoOmroFzVuQYkZuREllTMRpeJdEwK809931OeopmKV4mBZSIJKVAwFESDRa4cIgBDIcjlxQbOqPh1dE/+r6ho5/Grn62Hm2loaOf/vPGxgAyUgORsBoNsXRKc8/qlUWDrDBLY2TnU0CJiFxE8KwgW3CR7cyMjr4hGjv7aOiI9MAaznrf2Bl5Ntir+5voPG+q/ch5irPTRsMs0hM700Mb6a0VZ6eTlpIc42QKKBGRceCcG52hWFWac9FteweGo5cRR3pi0dfo14n2PrbVt9HcPcCFVqMryEp97+XEaKhNycmgIJRGUSiNvMzUuO6VeRpQzrmVwPeAIPDPZvYdL+sREZkMmWlBZhZlMbMo66LbDQ6Hae4aOKcndv7lxQMNXTR29Y9O+Lh2Rj5vHWsDIr2ygqxUCkNpFIbSKAqlj74vzk6jMPp9UXbkZwVZaQR9FGieBZRzLgj8CPgQUA+84Zz7DzPb7VVNIiJ+khoMMDUvcl/XxYTDRlvvIA2dfbR0DdDY1U9z1wAt3QM0dw/Q0t1PS/cAe0520Nw98J7lqEY4B/mZqaNhNhJcRdFQK8xOH31fFEqjIJRG6gROy/eyB7UM2G9mBwGcc/8KfBxQQImIXIJAwI32jGIxOBymtScSYC1dIyF2Jsyaoz97t6GLlu4BWnsufKkRIvetjQbYWb2xwlAkzIqy0855cvSl8DKgpgPHzvq+HljuUS0iIkkjNTgys/DiPbMRw2GjrefsEIu8Nnf1n/lZ1wBHmnt482gbrT3nTs0//J07L6tOLwPqQhc635PRzrnVwGqA1NRUampqJrquuNbY2EhJyeX930oyUTuNTW00NrXRhRUQCbWhsDEUDuMeXLXTzBZe6nG8DKh6YMZZ35cDJ87fyMweAR4BCIVCemDhGGpqalAbjU3tNDa10djURrFxzvVdzn5eTqZ/A7jKOVfhnEsD7gb+w8N6RETERzzrQZnZkHPuy8CzRKaZP2Zmu7yqR0RE/MXT+6DMbB2wLtbti4uLJ7CaxLB69WqvS4gLaqexqY3GpjaK2SOXs5Oz95s76EM1NTUagxIRiT+XdfdvcizoJCIicceXAeWcW+mce8c5t9859/XzPzcz/uRP/oSqqiqqq6t58803vSjTUxs2bODqq6+mqqqK73znvStEPfHEE1RXV1NdXc0NN9zAtm3bPKjSW2O10Yg33niDYDDIL3/5y0mszj9iaaeNGzeyZMkSFixYwC233DLJFXpvrDZqb2/nYx/7GIsXL2bBggU8/vjjHlTprXvvvZfS0lIWLrzwbHIX8f3o3/Xtzrnrxjyomfnqi8iEiQNAJZAGbAPmmxlLly41M7NnnnnGVq5caeFw2F577TVbtmyZJZOhoSGrrKy0AwcOWH9/v1VXV9uuXbvO2WbTpk3W0tJiZmbr1q1TG12gjUa2u+222+yjH/2o/eIXv/CgUm/F0k6tra02b948O3LkiJmZnT592otSPRNLG33729+2r371q2Zm1tDQYAUFBdbf3+9FuZ55+eWXbevWrbZgwYILfQxwB7CeyOW+WuB1GyMP/NiDGl0CycwGgJElkEY9/fTTfO5zn8M5R21tLW1tbZw8edKTYr2wZcsWqqqqqKysJC0tjbvvvpunn376nG1uuOEGCgoKAKitraW+vt6LUj0TSxsB/OAHP+CTn/wkpaWlHlTpvVja6cknn+QTn/gEM2fOBEi6toqljZxzdHZ2YmZ0dXVRWFhISkpyPSzi5ptvprCw8GKbfBz4aTSwNgP5zrlpF9vBjwF1oSWQpp+9wfHjx5kx48w9vuXl5Rw/fnxyqvOBS/39H330UT760Y9ORmm+EUsbHT9+nKeeeoo1a9ZMdnm+EUs77du3j9bWVm699VaWLl3KT3/608ku01OxtNGXv/xl9uzZQ1lZGYsWLeJ73/segYAf/7x6asy/7efzY8SPuQSSXWDmYTI9UvlSfv+XXnqJRx99lFdffXWiy/KVWNro/vvv58EHHyQYDE5WWb4TSzsNDQ2xdetWXnjhBXp7e1mxYgW1tbXMnTt3ssr0VCxt9Oyzz7JkyRJefPFFDhw4wIc+9CFuuukmcnNzJ6vMeBDT8nZn82NAjbkEUnl5OceOnQni+vp6ysrKJqc6H4j199++fTv33Xcf69evp6ioaDJL9FwsbVRXV8fdd98NQFNTE+vWrSMlJYW77rprMkv1VCztVF5eTnFxMaFQiFAoxM0338y2bduSJqBiaaPHH3+cr3/96zjnqKqqoqKigr1797Js2bLJLtfPYlre7hxjDVJN9heR0DwIVHBmksQCO2uSxNq1a8+ZJHH99ddfxpBe/BocHLSKigo7ePDg6KDtzp07z9nmyJEjNmfOHNu0aZNHVXorljY62z333JOUkyRiaafdu3fb7bffboODg9bd3W0LFiywHTt2eFTx5IuljdasWWPf/OY3zczs1KlTVlZWZo2NjR5U661Dhw5dbJLEnZw7SWKLjZEHvutB2fssgeScWzMySHvHHXewbt06qqqqyMrKSropnSkpKfzwhz/kIx/5CMPDw9x7770sWLCAhx9+GIA1a9bwrW99i+bmZr70pS+N7pNMNznH0kYSWzvNmzePlStXUl1dTSAQ4L777nvfqcSJKJY2+qu/+is+//nPs2jRIsyMBx98MOlWvvn0pz/Nxo0baWpqory8nAceeIDBwciDEaP/3tYRmcm3H+gBvjDWMbWShIiITDStJCEiIolDASUiIr6kgBIREV9SQImIiC8poERExJcUUCIi4ksKKBER8SUFlIiITIi77rqLpUuX4pzb5Zxbfan7+24lCRERSQyPPfYYhYWFOOdqgDecc/9uZs2x7u9JD8o59zvRRA1HCxcRkQTz/e9/n8WLFwNsJrJQ7FWXsr9Xl/h2Ap8AXvHo/CIiMoE2btzI888/z2uvvYaZLQbeAjIu5RieXOIzsz2QXM9wEhFJJu3t7RQUFJCVlYVz7hoiK5hfEk2SEBGRcbdy5UqGhoaorq4G+Bsil/kuyYT1oJxzzwNTL/DRN8zs6Us4zmpgNcDI4zZERMTf0tPTWb9+/ci3v3M5x5iwgDKz3xqn4zwCPAKRx22MxzFFRMT/dIlPRER8yatp5r/tnKsHVgDPOOee9aIOERHxL69m8T0FPOXFuUVEJD7oEp+IiPiSAkpERHxJASUiIr6kgBIREV9SQImIiC8poERExJcUUCIi4ksKKBER8SUFlIiI+JICSkREfEkBJSIivqSAEhERX1JAiYiILymgRETElxRQIiLiSwooERHxJQWUiIj4kgJKRER8SQElIiK+pIASERFfUkCJiIgveRJQzrn/4Zzb65zb7px7yjmX70UdIiLiX171oJ4DFppZNbAP+AuP6hAREZ/yJKDM7FdmNhT9djNQ7kUdIiLiX34Yg7oXWO91ESIi4i8pE3Vg59zzwNQLfPQNM3s6us03gCHgiYscZzWwGmDmzJkTUKmIiPiRMzNvTuzcPcAa4INm1hPLPjU1NVZXVzexhYmIyHhzl7PThPWgLsY5txL4GnBLrOEkIiLJxasxqB8COcBzzrm3nXMPe1SHiIj4lCc9KDOr8uK8IiISP/wwi09EROQ9FFAiIuJLCigREfElBZSIiPiSAkpERHxJASUiIr6kgBIREV9SQImIiC8poERExJcUUCIi4ksKKBER8SUFlIiI+JICSkREfEkBJSIivqSAEhERX1JAiYiILzkz87qGmDnnNpjZSq/rEBGRiRdXASUiIslDl/hERMSXFFAiIuJLCigREfElBZSIiPiSAkpERHzp/wNvK595AdItBQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sympy.plotting.plot(-sympy.log(a), xlim=(0,1));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If $y=0$: $L= - \\log(1-a)$. The plot again shows that the gradient is largest when the model output is furthest from the true value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEhCAYAAAAqDTTQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfsElEQVR4nO3deXRc5Z3m8efVLpX21ZYl2bIlG1m2DFgYA8FgTMBmSZgk0zGdTBaa47hzaCY5JIQsdALd6RN6Oj0DpCcZJoQzdEMISxgSMCbYIAcYAl7ANt43bEmWtUuWSkuVVO/8UZIsG9sqS1W6V9L3c04dVZVu3fr5PaDn/O5973uNtVYAALhNlNMFAABwNgQUAMCVCCgAgCsRUAAAVyKgAACuREABAFyJgAIAuBIBBQBwJQIKAOBKMSNtYIxJkHSLpKsl5UvqlvSRpFestbsiWx4AYKoy51vqyBjzE0m3SqqStFVSg6QESXMlLR94fo+1dkekCwUATC0jBdTN1tpXzvP7XElF1totkSgOADB1nTegAABwyojnoCTJGJMj6XuS5it4WE+SZK29LkJ1AQCmuFBn8T0laY+kYkkPSPpY0uYI1QQAQGiH+IwxW621i40xO6y1FQPvbbLWXhPxCodZuXKlXb9+/Xh+JQBg7MxoPhTSIT5J/oGfdcaYmyUdl1Qwmi8ci6ampvH+SgCAQ0INqH80xqRJukfSo5JSJX07YlUBAKa8kALKWvvywNN2Ba9/AgAgos47ScIY8yNjTOZ5fn+dMeaW8JcFAJjqRuqgdkr6ozGmR9I2SY0KTjMvlXSxpA2S/imSBQIApqaRppl/wVp7laTXJO2SFC3ppKT/kLTEWvtta23jaL/cGJNujHneGLPXGLPHGHPFaPcFAJhcRuqgFhtjZkr6kj557ilRwYVjx+JhSeuttV8wxsRJShrj/gAAk8RIAfUrSeslzZY0fL09I8kOvD8qxphUScskfU2SrLU+Sb7R7g8AMLmc9xCftfYRa22ZpN9Ya2cPexRba0cdTgNmK3hO6wljzAfGmF8bYzxj3CcAYJIIaakja+3fRuC7YyRdKumX1tpLJHkl3XfmRsaYNcaYLcaYLY2Noz7dBQCYYJy8o26NpBpr7XsDr59XMLBOY619zFpbaa2tzMnJGdcCAQCj0x+w6u3rH9M+HAsoa+0JSdXGmHkDb62QtNupegAA4XOgoUPzfrRer+6sG/U+Ql3qKFL+TtJTAzP4Dkv6usP1AADCoKUzOOctPSlu1PtwNKCstR9KqnSyBgBA+DV5gwGVlTz6gHLyHBQAYJJq6eyVJGV6CCgAgIu0eH0yRsoYwyE+AgoAEHbNXp8ykuIUHTWqexVKIqAAABHQ3Okb0+E9iYACAERAi5eAAgC4ULO3V9ljmMEnEVAAgAiggwIAuE5ff0CtXX5leuLHtB8CCgAQVq1dfklSFh0UAMBNWsKwioREQAEAwqzZO/ZVJCQCCgAQZs0DC8VmcQ4KAOAmg4f46KAAAK7SPLQOX+yY9kNAAQDCqsXbq/TEWMVEjy1iCCgAQFiF4yJdiYACAIRZU6dvzBMkJAIKABBmLV7fmK+BkggoAECYcYgPAOA6/QGr1i7fmJc5kggoAEAYtXb5ZO3Yr4GSCCgAQBidWoePSRIAABc5tcwRHRQAwEWGljliFh8AwE3CtZK5REABAMJo8BBfZhIBBQBwkRavT+lJY1+HTyKgAABhFK6LdCUCCgAQRk2dvWGZwScRUACAMGrxhmehWImAAgCEUYvXF5Yp5hIBBQAIk3CuwycRUACAMGnr8ikQpnX4JAIKABAm4VyHTyKgAABh0uwN3zp8EgEFAAiToXX4CCgAgJs0dwbX4aODAgC4yuAhvgwCCgDgJi1en9ISYxUbhnX4JAIKABAmzd7wXQMlEVAAgDBp7uwN2wQJiYACAIRJOFcylwgoAECYtHh9YbtIVyKgAABh0N7l1/S0BFUUpIZtnwQUAGDMqvY3aGftSc3NI6AAAC6yYU+DspPjdHFhetj2SUABAMbE3x9Q1b4GLZ+Xq+goE7b9ElAAgDHZfKRFHT19un5+Xlj3S0ABAMZkw54GxcVE6erS7LDu1/GAMsZEG2M+MMa87HQtAIALY63Vxr31unJOlpLiYsK6b8cDStJ/lbTH6SIAABfuYEOnjjZ36fqy8B7ekxwOKGNMgaSbJf3ayToAAKOzYU+DJGlFWW7Y9+10B/U/JN0rKeBwHQCAUdiwp17l+amanpYY9n07FlDGmFskNVhrt46w3RpjzBZjzJbGxsZxqg4AMJLmzl5tO9YakcN7krMd1FWSPmOM+VjSM5KuM8b8x5kbWWsfs9ZWWmsrc3JyxrtGAMA5vLmvUdZq8gWUtfb71toCa+0sSaslvWGt/bJT9QAALsyG3fXKS43XghnhW95oOKfPQQEAJqDevn69daBRK8ryZEz4Vo8YLryT1kfJWlslqcrhMgAAIfrL4RZ5ff26PgKz9wbRQQEALtiG3fVKjI3WlXPCu3rEcAQUAOCCWGu1cU+9PlWarYTY6Ih9DwEFALgge+o6dLy9J6KH9yQCCgBwgTbsqZcx0nUXRWZ6+SACCgBwQTbuqdeignTlpMRH9HsIKABAyOpP9mh7TXvED+9JBBQA4AK8sTe4OGy4b054NgQUACBkG/fUa0Z6oublpUT8uwgoAEBIun39eutAk64vy43Y6hHDEVAAgJC8c7BJvX2BcTm8JxFQAIAQbdxbr+T4GF1enDUu30dAAQBGFAhYbdjToGvm5iguZnyig4ACAIxoZ227Gjt6I3Jr93MhoAAAI9q4p15RRlo+j4ACALjI63saVDkzUxmeuHH7TgIKAHBetW3d2lN3clwP70kEFABgBC9vPy5JWlE2PtPLBxFQAIBzaujo0WN/PqzblxRqTo5nXL+bgAIAnFUgYHXPs9vl9fXp61cVj8vqEcMRUACAs/pffz6stw406e9vKdfccVh770wEFADgE7Yda9W//Gmfbl44XbcvKXSkBgIKAHCa9m6/7v7tB5qWmqB/+tzCcT+0NyjGkW8FALiStVY/+P1O1bX36Lm1VygtMdaxWuigAABDntlcrVd21umeG+bq0qIMR2shoAAAkqT99R36yR926VMl2Vq7bI7T5RBQAACpx9+vu57eppSEGP3rFxcpKsqZ807DcQ4KAKAHX96t/fWdevKOJcpNSXC6HEl0UAAw5a3bWaen3zumb1wzW8vm5jhdzhACCgCmsOqWLn3vhR1aVJiu79wwz+lyTkNAAcAU5e8P6O5nPpCs9OjqSxQb7a5I4BwUAExR//31/frgWJsevf0SFWUlOV3OJ7grLgEA4+LtA0365aZDWn1ZoW5dlO90OWdFQAHAFNPY0atvP/uh5uQk68e3ljtdzjlxiA8AppBAwOqe57brZLdf//43S5QYF+10SedEBwUAU8iv3z6sP+9v1P23zNdF01KdLue8CCgAmCI+rG7TP6/fp1ULpulLlxc5Xc6ICCgAmAJO9vj1d7/dprzUBP3scxWO3ULjQnAOCgAmuUDA6r4Xduh4W4+e/cZSpSU5dwuNC0FAAcAkVt3SpXuf36Eef7++v+oiLZ6Z6XRJISOgAGASstbqmc3V+seXd8sYox/dXKYvXubMrdtHi4ACgEmmrr1b972wU5v2N+rKOVn65y9UqCDDfStFjISAAoBJwlqr32+r1U/+uEt9/VYPfrZcX758pivu7TQaBBQATAINHT364Ysf6fXd9aqcmaF/+c+LNCvb43RZY0JAAcAE9/KO47r//34kr69fP7ypTHd8qljRE7RrGo6AAoAJqsXr0/0vfaRXdtRpUUGafv5Xi1SSm+J0WWFDQAHABPSnXSf0gxd3qr3br+/eOE/fWDZbMS67n9NYEVAAMIG0d/n1wB936fcf1Gr+9FT9+99crrLp7l5Tb7QIKACYIKr2Neh7L+xQU6dPd68o1V3LSxQXM7m6puEIKABwuc7ePv30ld367fvVKs1N1v/+SqUqCtKdLiviHAsoY0yhpCclTZMUkPSYtfZhp+oBADf6f4ea9N3ndqiuvVvfuGa2vn39XCXEuvceTuHkZAfVJ+kea+02Y0yKpK3GmNettbsdrAkAXKHL16eHXt2r//PuURVne/Tc2iu1eGaG02WNK8cCylpbJ6lu4HmHMWaPpBmSCCgAU9qWj1v0nee26+PmLn39qlm698aLXH3n20hxxTkoY8wsSZdIes/hUgDAMT3+fv38T/v067ePqCAjUc+sWaqls7OcLssxjgeUMSZZ0guSvmWtPXmW36+RtEaSiorcfwdIABiND6vbdM+zH+pQo1d/fXmRfnBTmZLjHf8T7ShjrXXuy42JlfSypNestf860vaVlZV2y5YtkS8MAMaJry+gRzYe0C83HVJuSrwe+nyFls3NcbqscBvVuktOzuIzkh6XtCeUcAKAyWbX8Xbd8+x27T3RoS8sLtD9t8xXWuLEuNvteHCyf7xK0n+RtNMY8+HAez+w1q5zriQAiDx/f0C/rDqkRzYeUIYnTo9/tVIryvKcLst1nJzF97ZG2fYBwETUH7Cq2tegx946rPcOt+gzi/L1wGfKleGJc7o0V5raZ+AAYBwcb+vWs1uq9bvN1apr79EN8/P0P790qW5aON3p0lyNgAKACBjslp5+75je3NcgK+nq0hz9+Nb5WlGWp9hJtvJ4JBBQABBGde3d+t3mU91STkq8/vbaOVp9WZEKM5OcLm9CIaAAYIwGu6Xfvn9Mb+xtUMBKV5dm0y2NEQEFAKN0ZreUnRyvtdcEu6WiLLqlsSKgAOAC9AesNu0Pnlsa3i39/S3zdf18uqVwIqAAIASD3dKzm6t1nG5pXBBQAHAO5+qW7qdbGhcEFACcoa69W89urtHvNh+jW3IQAQUAGt4tVeuNvfWndUsryvIUF0O3NN4IKABTWk1rl17YWjusW4rTN66Zo9vplhxHQAGYUvr6A9p2rE1V+xr05r5G+fr6dajRq6tLs/WjW+brerol1yCgAEx6DR092rSvUVX7GvXWgUad7OlTdJRR5cwMLZ83XTdX5LPKgwsRUAAmnf6A1YfVrXpzb6Oq9jfoo9rgzbpzU+K1csE0LZ+Xq6tKs5WawL2X3IyAAjApNHX2atO+Rr25r0FvHWhSe7df0VFGlxal67s3ztO183I0f3qqgvdKxURAQAGYkPoDVttr2lS1r1FV+xq0o6ZdkpSdHK9Pz8/TtfNydHVJjtKS6JImKgIKwITR4vXpz/uDXdKf9zeqtcuvKCNdUpSh79wwV9fOy9X86amKiqJLmgwIKACuFQhY7ahtH5pxt6OmTdZKWZ44Lb8oV9fOy9Wy0mylJ3FH2smIgALgKs2dvXr7YJM27WvUpv2Navb6ZIy0qCBd31oxV9fOy9HCGWl0SVMAAQXAMYGA1eGmTm092jr0iI2K0t76DmV64rSsNFvLL8rV1aU5yvTQJU01BBSAcdPl69OH1W3aNhBG2461qb3bL0lKT4rV4qIMLSnO1JLiTFUUpCuaLmlKI6AARIS1VrVt3cEgOtqqrcdataeuQ/0BK0kqzU3WqgXTdOnMDC2emaHZ2R6mgOM0BBSAsPD1BbS77qS2fNyibceCHVL9yV5JUmJstC4uTNc3r52jS2dm6JLCdCY2YEQEFIBRae7s1bZjbUMd0vaaNvX2BSRJM9ITdXlxlhYPdEcXTUtRDPdOwgUioACMqMffr/31Hdp/okPvHg52SEeavJKk2Gij8vw0fXnpTC2emaFLizI0LS3B4YoxGRBQAE7T3u3X7uMntet4u3bXndTu4yd1sKFTfQGrpcWZOtDQqUtnZuiLlxVq8cwMLZyRpoTYaKfLxiREQAFTlLVW9Sd7tet4u3YNC6Tqlu6hbXJT4lWen6rry/I0Pz9V5fmpKspMYjIDxgUBBUwBgYDVkWbvqSA6HuyMmr2+oW2Ksz2qKEjX7UuKNH96qsrz05STEu9g1ZjqCChgkunx9+tAfedpndHeEx3q8vVLCp4zmpuXohVluSrPT9P8/FSVTU9Vcjx/DuAu/BcJTFA9/n4dauzUwYZOHagf+NnQoazkeL1/pEWSlBwfo/nTU/VXlYUqz0/V/PxUleamcMdYTAgEFOBynb19OtTQqQMDATT4/FhLl2zwmldFRxnNzEpSaW6yFs/M1NeunKXy/FQVZiSxZh0mLAIKcIn2Lr8ONnboQP1gGHXqUEOnattOTVqIjTaanZ2sBTPS9J8umaHS3BSV5iVrZlaS4mOYSYfJhYACxllzZ+9QAB2s7wj+bOhUQ0fv0DYJsVGak5Osy2Zl6K/zilSSm6zS3GQVZSZxwSumDAIKiAB/f0A1rd36uMmrE+3d2jlwLdHBhk61DJs554mLVkleipbNzVFpbrJK85JVmpuiGemJHJrDlEdAAaPUNxBCR5q9Otrk1cfNXTrS5NXHzV7VtHYPLYo6f3qqatu6VZqbrBvL81SSmzIURtNSE7imCDgHAgo4j77+gI639ehIs1cfN3l1pMmro83BMKpu6VLfQAhJwW5oVrZHC/LTdEvFdM3K8qg426OirCTlJMcTRMAFIqAw5XX7+lXT2qXq1i7VtvXoSGOwC/q42avqli75+0+FUFJctGZmeVQ2PUWrFkzTrGyPZmV5NCubEALCjYDCpNfXH1Bde4+qW4IhVN3SPfCzS9Wt3WocNjkhJkpKiI1RQUai5uWl6Ib501ScnTTUDeWkEELAeCGgMOFZa9XY2avqlu5gJ9QyLIRau3S8rWfofJAUvGZoelqCCjOStHxejgozklSYOfhIVLYnTlFRzJQDnEZAwfWstWr2+nS8rVv1J3t0tPlU91Pd0qWa1m51+/tP+0x2crwKMxN1SWGGPrMoUYUZSSoaCKFpaQmKZao24HoEFBzX5evT8bYeHW/rDj7ahz0feO0buBFeSa5HBxu8SomPUUFmkoqzPVo2N0eFGYkqzAyGUEFGkhLjuGgVmOgIKERUf8CqoWMwcE4FT+3A87r2brV2+U/7jDFSXkqC8tMTVD4jTTeUT1N+WoLy0xOVn56ggowkpSXGci4ImOQIKIxaIGDV0uXTifYeNXX2qrq1+/TOp61HJ06efv5HklISYjQjPVH56Ym6dGa6pqclDr3OT09QXiqH4AAQUDiHHn+/6k/26ER7MGSCz3uDPwfeb+joGZqCvbgoQ1uPtSo22mhaWoLy0xJ1eXGm8tMTNT092P3MSE/U9LQEpSTEOvyvAzAREFBTzPCuZzBs6gdC6MTJXjUMvNd2xmE3KXgN0LTUYIezpDhTeakJmpYar2lpCZqWGgyh7OR4lugBEBYE1CQRCFi1dvnU2NmrhpO9auzoVUNHrzp6/Dra3HXWrmeQMcFZb9NSg+d3KmdlDAXRYPjkpSUoJT6G8z4Axg0B5XK9ff1DYTP8Z2NHz9DrhpO9aursPW3ZnUF5qfHyxMWctesZDKDs5HjO+QBwHQLKAdZanezuU2NnT7DbGeh6GoaFzuDP9u5PHmozRsryxCsnJV65KfGal5cy9DwnJUG5qfHKSQ7+3sNtvAFMUPz1CpO+/oBau/xq6uxVc6dPzd5eNXX6Bl4H32vq7FV7j1/H205d1zNcfEyUclPjlZuSoDk5ybpiTpZykuOH3hsMoUxPHPcEAjDpORpQxpiVkh6WFC3p19banzlZz5m6fH1q6vCpyXsqYJo7hwfPqSBq7fIN3X57uJgoo6zkOGV54pWdEq85OclaVZ6grOS4gcAZCJ7UeM7xAMAwjgWUMSZa0r9J+rSkGkmbjTF/sNbujtR39vcH1NrtDwZLZ/DQ2lDIdJzZ9fg+sXzOoJSEGGUnxyvLE6fibI8um5WprOR4ZSfHDb0/+JoLSgFgdJzsoJZIOmitPSxJxphnJH1WUkgB1R+wOtntV0uXT61en1q7/Gr1+oKvB95r8fqDzwdeZyfH60BD5yf2FR1lTguV4myPsjxxyk4Jhk12cnwweJLjlOmJU0Isy+gAQKQ5GVAzJFUPe10j6fLzfeBQY6eu+3mVWr0+tXX7z3pITZLiYqKUmRSnDE+cMj2xmj89VRlJcZqWmqDkwe4nOW6o40lNiOXaHQBwGScD6myJ8InIMcaskbRGkqLiEvTRv31TMVFG0QOPmKioU6+jjWKijKKMUZ+kxoHHVNLY2KicnByny3A9xmlkjNHIGKPQbN269SNr7YIL/ZyTAVUjqXDY6wJJx8/cyFr7mKTHJMnj8diGwxE7RTUpVFZWasuWLU6X4XqM08gYo5ExRqExxvSM5nNOzlXeLKnUGFNsjImTtFrSHxysBwDgIo51UNbaPmPMXZJeU3Ca+W+stbucqgcA4C6OXgdlrV0naV2o22dnZ0ewmslhzZo1TpcwITBOI2OMRsYYheyx0XzI2HNNhXOhyspKy/FeAJhwRjVNmvVyAACu5MqAMsasNMbsM8YcNMbcd+bvrbW6++67VVJSooqKCm3bts2JMh21fv16zZs3TyUlJfrZzz65QtRTTz2liooKVVRU6Morr9T27dsdqNJZI43RoM2bNys6OlrPP//8OFbnHqGMU1VVlS6++GKVl5frmmuuGecKnTfSGLW3t+vWW2/VokWLVF5erieeeMKBKp11xx13KDc3VwsWnH02uQl6ZODv+g5jzKUj7tRa66qHghMmDkmaLSlO0nZJ8621Wrx4sbXW2ldeecWuXLnSBgIB++6779olS5bYqaSvr8/Onj3bHjp0yPb29tqKigq7a9eu07Z55513bEtLi7XW2nXr1jFGZxmjwe2WL19uV61aZZ977jkHKnVWKOPU2tpqy8rK7NGjR6211tbX1ztRqmNCGaOf/vSn9t5777XWWtvQ0GAzMjJsb2+vE+U6ZtOmTXbr1q22vLz8bL+WpJskvarg4b6lkt6zI+SBGzuooSWQrLU+SYNLIA156aWX9JWvfEXGGC1dulRtbW2qq6tzpFgnvP/++yopKdHs2bMVFxen1atX66WXXjptmyuvvFIZGRmSpKVLl6qmpsaJUh0TyhhJ0qOPPqrPf/7zys3NdaBK54UyTk8//bQ+97nPqaioSJKm3FiFMkbGGHV0dMhaq87OTmVmZiomZmrdLGLZsmXKzMw83yaflfTkQGD9RVK6MWb6+T7gxoA62xJIM4ZvUFtbq8LCU9f4FhQUqLa2dnyqc4EL/fc//vjjWrVq1XiU5hqhjFFtba1efPFFrV27drzLc41Qxmn//v1qbW3Vtddeq8WLF+vJJ58c7zIdFcoY3XXXXdqzZ4/y8/O1cOFCPfzww4qKcuOfV0eN+Lf9TG6M+BGXQLJnmXk4lVYMv5B//5tvvqnHH39cb7/9dqTLcpVQxuhb3/qWHnroIUVHT93Ff0MZp76+Pm3dulUbN25Ud3e3rrjiCi1dulRz584drzIdFcoYvfbaa7r44ov1xhtv6NChQ/r0pz+tq6++WqmpqeNV5kQQ0vJ2w7kxoEZcAqmgoEDV1aeCuKamRvn5+eNTnQuE+u/fsWOH7rzzTr366qvKysoazxIdF8oYbdmyRatXr5YkNTU1ad26dYqJidFtt902nqU6KpRxKigoUHZ2tjwejzwej5YtW6bt27dPmYAKZYyeeOIJ3XfffTLGqKSkRMXFxdq7d6+WLFky3uW6WUjL251mpJNU4/1QMDQPSyrWqUkS5XbYJImXX375tEkSl1122ShO6U1cfr/fFhcX28OHDw+dtP3oo49O2+bo0aN2zpw59p133nGoSmeFMkbDffWrX52SkyRCGafdu3fb6667zvr9fuv1em15ebnduXOnQxWPv1DGaO3atfbHP/6xtdbaEydO2Pz8fNvY2OhAtc46cuTI+SZJ3KzTJ0m8b0fIA9d1UPYcSyAZY9YOnqS96aabtG7dOpWUlCgpKWnKTemMiYnRL37xC914443q7+/XHXfcofLycv3qV7+SJK1du1YPPvigmpub9c1vfnPoM1PpIudQxgihjVNZWZlWrlypiooKRUVF6c477zznVOLJKJQxuv/++/W1r31NCxculLVWDz300JRb+eb2229XVVWVmpqaVFBQoAceeEB+v1/S0P9v6xScyXdQUpekr4+0T1aSAABEGitJAAAmDwIKAOBKBBQAwJUIKACAKxFQAABXIqAAAK5EQAEAXImAAgBExG233abFixfLGLPLGLPmQj/vupUkAACTw29+8xtlZmbKGFMpabMx5gVrbXOon6eDAgBExCOPPKJFixZJ0l8UXCi29EI+T0ABAMKuqqpKGzZs0Lvvvitr7SJJH0hKuJB9EFAAgLBrb29XRkaGkpKSZIy5SMEVzC8IAQUACLuVK1eqr69PFRUVkvQPCh7muyCsZg4AiDRWMwcATB4EFADAlQgoAIArEVAAAFcioAAArkRAAQBciYACALgSAQUAcCUCCgDgSgQUAMCVCCgAgCsRUAAAVyKgAACuREABAFyJgAIAuBIBBQBwJQIKAOBKBBQAwJUIKACAKxFQAABXIqAAAK5EQAEAXImAAgC4EgEFAHAlRwLKGPPfjDF7jTE7jDEvGmPSnagDAOBeTnVQr0taYK2tkLRf0vcdqgMA4FKOBJS19k/W2r6Bl3+RVOBEHQAA93LDOag7JL3qdBEAAHeJidSOjTEbJE07y69+aK19aWCbH0rqk/TUefazRtIaSSoqKopApQAANzLWWme+2JivSloraYW1tiuUz1RWVtotW7ZEtjAAQLiZ0XwoYh3U+RhjVkr6nqRrQg0nAMDU4tQ5qF9ISpH0ujHmQ2PMrxyqAwDgUo50UNbaEie+FwAwcbhhFh8AAJ9AQAEAXImAAgC4EgEFAHAlAgoA4EoEFADAlQgoAIArEVAAAFcioAAArkRAAQBciYACALgSAQUAcCUCCgDgSgQUAMCVCCgAgCsRUAAAVyKgAACuREABAFyJgAIAuJKx1jpdQ8iMMeuttSudrgMAEHkTKqAAAFMHh/gAAK5EQAEAXImAAgC4EgEFAHAlAgoA4Er/H1ClvPts5vN/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sympy.plotting.plot(-sympy.log(1-a), xlim=(0,1));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find the parameters using `autograd`\n",
    "\n",
    "With the logistic loss function, we're ready to use gradient descent in an optimization loop. \n",
    "Above, we defined the `logistic` SymPy expression as a function of the symbol `z`, so let's re-define it to include the composition with $wx+b$. \n",
    "We could then define the loss function symbolically, as we did for linear regression; have a look at this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{1}{1 + e^{- z}}$"
      ],
      "text/plain": [
       "1/(1 + exp(-z))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logistic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle - y \\log{\\left(\\frac{1}{e^{- b - w x} + 1} \\right)} - \\left(1 - y\\right) \\log{\\left(1 - \\frac{1}{e^{- b - w x} + 1} \\right)}$"
      ],
      "text/plain": [
       "-y*log(1/(exp(-b - w*x) + 1)) - (1 - y)*log(1 - 1/(exp(-b - w*x) + 1))"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w, b, x, y = sympy.symbols('w b x y')\n",
    "logistic = 1/(1+ sympy.exp(-w*x-b)) # redefined with the composition\n",
    "\n",
    "Loss = -y*sympy.log(logistic) - (1-y)*sympy.log(1-logistic)\n",
    "Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we can still use SymPy to get derivatives of the loss function with respect to the parameters. But with more complicated models, finding symbolic derivatives could take a long time.\n",
    "\n",
    "Have a look at the derivative of the logistic loss with respect to the parameter $b$: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle - \\frac{y e^{- b - w x}}{e^{- b - w x} + 1} - \\frac{\\left(y - 1\\right) e^{- b - w x}}{\\left(1 - \\frac{1}{e^{- b - w x} + 1}\\right) \\left(e^{- b - w x} + 1\\right)^{2}}$"
      ],
      "text/plain": [
       "-y*exp(-b - w*x)/(exp(-b - w*x) + 1) - (y - 1)*exp(-b - w*x)/((1 - 1/(exp(-b - w*x) + 1))*(exp(-b - w*x) + 1)**2)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Loss.diff(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we can use symbolic differentiation, we later need to convert the resulting expression to a Python function that can be called and evaluated at many data inputs. Maybe this is not the best approach.\n",
    "\n",
    "There's a better way! It's called _automatic differentiation_: the idea is to algorithmically obtain derivatives of numeric functions written in computer code. \n",
    "It sounds like magic, but it can be done by a combination of the chain rule, symbolic rules of differentiation for elementary operations, and a numeric evaluation trace of the elementary derivatives. \n",
    "\n",
    "The details of how automatic differentiation works are fascinating, but here we skip ahead to say that software libraries are freely available to do this! \n",
    "For Python programs, the [`autograd`](https://github.com/hips/autograd) package was specifically developed for gradient-based optimization. \n",
    "It is described thus:\n",
    "\n",
    "> Autograd can automatically differentiate native Python and NumPy code. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can even take derivatives of derivatives of derivatives.\n",
    "\n",
    "(The package is no longer in active development, as a new version was folded into the [JAX](https://github.com/google/jax) project, out of Google. It is still maintained, though, and just as useful as ever!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To install the package, you can run the following command (either on the terminal or in a Jupyter code cell):\n",
    "\n",
    "```\n",
    "pip install autograd\n",
    "```\n",
    "\n",
    "Or, if you installed your Python environment via [Anaconda](https://www.anaconda.com/products/individual), you can run this command:\n",
    "```\n",
    "conda install -c conda-forge autograd\n",
    "```\n",
    "\n",
    "Once the package is installed in your working environment, you can import it in the usual Pythonic way. \n",
    "The main function you will need is `autograd.grad()`, which takes a scalar-valued Python function as argument, and returns another function that evaluates to its derivative. \n",
    "It's what we use in the optimization loop to perform gradient descent.\n",
    "\n",
    "In addition, `autograd.numpy` is a wrapper to the NumPy library. This allows you to call your favorite NumPy methods with `autograd` keeping track of every operation so it can give you the derivative (via the chain rule)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the autograd-wrapped version of numpy\n",
    "from autograd import numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the gradient calculator\n",
    "from autograd import grad  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's important to realize that we now need to design our program in a _functional_ style: defining Python functions for the logistic and loss functions, so that we may apply `autograd.grad()` in the optimization loop. \n",
    "Study the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note: the namespace numpy is the autograd wrapper to NumPy\n",
    "\n",
    "def logistic(z):\n",
    "    '''The logistic function'''\n",
    "    return 1 / (1 + numpy.exp(-z))\n",
    "    \n",
    "def logistic_model(params, x):\n",
    "    '''A prediction model based on the logistic function composed with wx+b\n",
    "    Arguments:\n",
    "       params: array(w,b) of model parameters\n",
    "       x :  array of x data'''\n",
    "    w = params[0]\n",
    "    b = params[1]\n",
    "    z = w * x + b\n",
    "    y = logistic(z)\n",
    "    return y\n",
    "\n",
    "def log_loss(params, model, x, y):\n",
    "    '''The logistic loss function\n",
    "    Arguments:\n",
    "       params: array(w,b) of model parameters\n",
    "       model:  the Python function for the logistic model\n",
    "       x, y:   arrays of input data to the model'''\n",
    "    y_pred = model(params, x)\n",
    "    return -numpy.mean(y * numpy.log(y_pred) + (1-y) * numpy.log(1 - y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get a function to compute the gradient of the logistic loss\n",
    "gradient = grad(log_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "function"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(gradient)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Note:\n",
    "\n",
    "> The argument list of the function returned by `grad()` will be the same as the argument list of the loss function that we input to it.\n",
    "> The default setting of `grad()` is to return the derivative(s) with respect to the first argument, in this case, the two-element array `params`. This gives us the two derivatives (with respect to $w$ and $b$) at once.\n",
    "\n",
    "Let's now make a random starting guess for the two model parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.5488135  0.71518937]\n"
     ]
    }
   ],
   "source": [
    "numpy.random.seed(0)\n",
    "params = numpy.random.rand(2)\n",
    "print(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With values assigned to the two parameters, this is how you compute the corresponding derivatives—we'll need to do this in each iteration of the optimization loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.42734877,  0.08274066])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradient(params, logistic_model, x_data, y_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we optimize! Notice that in this optimization loop we chose to use a `while` statement, instead of `for`. \n",
    "We set both a maximum number of iterations, and an exit criterion based on the norm of the gradient being very small.\n",
    "Already you should be thinking about these choices we make:\n",
    "\n",
    "- The gradient is multiplied by a small step of $0.01$: this is called the _learning rate_. How do we choose this value?\n",
    "- The `while` loop exits when the norm of the gradient is $0.001$: how do we know if this is a good choice?\n",
    "\n",
    "Finally, we plot the synthetic data we created above, and the logistic regression curve corresponding to the parameters found in the optimization loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimized value of w is 2.731 vs. true value: 2\n",
      "Optimized value of b is 0.697 vs. true value: 1\n",
      "Exited after 5000 iterations\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeVklEQVR4nO3de3SV9Z3v8fc3d0JuQAIooFzEC/VSJQWqeGqtR/Eypdep1ku17bic0U47PZ1W29VO53J6GU+PdtQu6njp6XRGawdqrYtitZMWQ7QaLxVRcRDkKhAIJIRkJ9nZ3/PHs4EYQrKT7OTZ+9mf11qu/PbeT5792bj5rIffczN3R0REsl9e2AFERCQ9VOgiIhGhQhcRiQgVuohIRKjQRUQioiCsN66urvaZM2eG9fYiIlnphRde2OPuNf29Flqhz5w5k8bGxrDeXkQkK5nZ5mO9pikXEZGIUKGLiESECl1EJCJU6CIiEaFCFxGJiEGPcjGzB4ArgN3ufno/rxvwQ+AyoB243t1fTHdQkf5samqj/q297GqJMaWyhMVzJjGrpmzIy/d+Pj8PDIgnGPJ4SmUJMyeM4+19HSNe12iPMz1rpucbadZUvq9DZYNdbdHM/gfQBvz0GIV+GfAFgkJfCPzQ3RcO9sa1tbWuwxZlJDY1tfHzxm1UjSukrKSAtlic/R3dfKp2er9/SY61/HmzJ7JmYzNV4wqJdcd5dmMzDpw2tYw3dralPD53zkQOxnp4bvM+FsycQGlR/rDXNdrjTM+a6flGmvXcORMpLigY8Pt6LGb2grvX9vtaKpfPNbOZwOPHKPQfA79394eSj9cDF7j7OwOtU4UuI/Vvz27mYCxOxbjCw8+1dnQzvqSAaxedmPLyr+5o4fTjK6kYV8izG/fSGU9gwPaWdo6vLE15XFSYBx6ss6K0EJxhr2u0x4eyHjgYoyYvTmFnDO/ooLirk+Z9rRxXkk9BvJv9LW1MLsmnoCfO/gPtVJcUkN/TQ0tbO5NKCshLJGhpjzEhOW5t76SypDAYd3RSVVyAudMa66KyuABzDo8BDsS6qDg87qa8uBADCpKTwbHuHkqSD3oSHqyr88hyvccHYt2UlxQeWVeax22xLsoOvVdn9+Fxfr4l/1/3UFyYDw7xhB+1XO/x5nlnU7X0igG/r8cyUKGn48SiacDWXo+3JZ87qtDN7EbgRoATTjghDW8tuWxXS4yplSXveq6spICdLbEhLf9OSweLZk8CoDXWTWVJIRi0tHcze1Je6uOObgAqxhXQmhwPe13DHBd3tBF76w3OLuqkct9u2t/ewlyLMf7AfhJNTUxJxBjXfoDig20Ud7ZT3NWZ7v8tkZYwS8t6VnZfS+vSKwb8vg5HOgq9v0/Y72a/u98L3AvBFnoa3lty2JTKEtr6bHG3xeJM6VPagy1/XOW4w89XlBQSS25VV5YObVwxrvCoLfThrmuw8biONs7b/ipnrd3KjC1vctPWt5i2ZwcVB/Yd9bkPlpbTVj6BfePKaamoYtfk6XSOL6ezpJTW/BKsfDxdRSW0FxQTLyxiVxeUV5bRU1DIzlgPEyrLSOQX8E5HnEmV4/G8PHYc7Kbm8LiLyRWlkJfHjtYYk6vGg+Wx40AHUyrHY2Zsb+1gauV4zGB7awfHVY4HM3a0tB95vqWd46rG9/oXhB3+s3TsyL92WjtC/xfOSP9lVlSYx6JBvq/DkY5C3wbM6PV4OrAjDesVGdDiOZP4eeM2gHfNiV/ynilDWv5T86ezZmMzALOrSw/Pf86fUXV4zjOV8aG51C3N7Zw8pexdc6lDXVff8Vtb9nL6G41cv+VlJr/SyMyt/02eJwBoKa+i+cS5PD/tfHbWTKNozmxeyatgb1UNx809kXV7O4c879vQa9735f7mkE/pNZ7Vazxv6pHxyZOCscPpJx13+PkzZ09Jjp2zZtUcfv7smdVH8k2r5GCsh837Y8w9riKtf5bpHg/n//u5cybS2tE94Pd1ONIxh345cAtHdor+i7svGGydmkOXdIjyUS5F8S5OaniKk1avYmZjPUWxdrqLS3jn1LPYccZ8tp16Ns1zTqF9Yg1xt8gdRZLp+cI6ymVEO0XN7CHgAqAa2AX8HVAI4O7Lkoct3g0sIThs8QZ3H7SpVegix7BpE/z4x/DAA9DUBMcfD0uXwkc+Ah/4ABQXh51QQjSinaLuftUgrztw8zCzicghO3bAP/wD3H8/uMOf/Rn85V/CRRdBns4BlMGFdvlcEUmKxYIiv+MOiMfhxhvhtttg+vSwk0mWUaGLhKmxEa67Dl5/Ha6+Oij22bPDTiVZSv+OEwlDIgF///ewaBG0tsITT8DPfqYylxHRFrrIWIvF4Npr4T//E665Bu66C6qqwk4lEaBCFxlLe/cGR6ysWQM/+AH8zd9Ams4+FFGhi4yVnTuDww43b4ZHHoFPfjLsRBIxKnSRsdDaCpddBtu2wZNPwvnnh51IIkiFLjLaurrgYx+DV16BX/9aZS6jRoUuMprc4YYb4He/gwcfhEsvDTuRRJgOWxQZTXfdBf/xH/BP/wTXXx92Gok4FbrIaFm7Fr76VbjiCvj618NOIzlAhS4yGmKx4MzPysrg2iw6NFHGgObQRUbDbbcFW+grV8LkyWGnkRyhLXSRdFu9Gu68E77wBe0ElTGlQhdJp54e+NKXYMYM+N73wk4jOUZTLiLp9NOfwksvBUe2lJaGnUZyjLbQRdLlwIHgaJZFi+DKK8NOIzlIW+gi6fK97wXXa3n0UR3VIqHQFrpIOmzeHFw98eqrYeHCsNNIjlKhi6TD7bcHN634znfCTiI5TIUuMlJNTfDAA8FNK044Iew0ksNU6CIjdffd0NEBX/lK2Ekkx6nQRUbi4MGg0JcuhdNOCzuN5DgVushI3H8/NDcHF+ESCZkKXWS4uruDI1sWL4Zzzw07jYiOQxcZtuXLYcuWYMpFJANoC11kuO67D2bNgssvDzuJCKBCFxmeTZuC28rdcAPk6a+RZAZ9E0WG4yc/CU7v/8xnwk4icpgKXWSoenqCGz5ffLFOJJKMokIXGar/+i/YuhU++9mwk4i8S0qFbmZLzGy9mW0ws1v7eb3SzH5tZn8ys3VmdkP6o4pkiPvvh4kTg5OJRDLIoIVuZvnAPcClwDzgKjOb12exm4HX3P0s4ALgB2ZWlOasIuFrboZf/hKuuQaKi8NOI/IuqWyhLwA2uPtGd+8CHgb6bpo4UG5mBpQBzUA8rUlFMsFDD0FXl6ZbJCOlUujTgK29Hm9LPtfb3cBpwA5gLfBFd0/0XZGZ3WhmjWbW2NTUNMzIIiH6xS9g3jw466ywk4gcJZVC7+/WK97n8SXAy8DxwHuBu82s4qhfcr/X3WvdvbampmaIUUVCtmsXPP00fOITYScR6Vcqhb4NmNHr8XSCLfHebgBWeGADsAk4NT0RRTLEo48GN7H4+MfDTiLSr1QK/XlgrpnNSu7ovBJ4rM8yW4APAZjZFOAUYGM6g4qEbvlyOOkkOOOMsJOI9GvQQnf3OHAL8ATwOvCIu68zs5vM7KbkYv8InGtma4HfAV9z9z2jFVpkzO3dGxx//olP6AbQkrFSutqiu68EVvZ5blmv8Q7g4vRGE8kgjz0WnCGq6RbJYDpTVCQVy5fDiSfC/PlhJxE5JhW6yGBaWuC3vw22zjXdIhlMhS4ymMcfD+5OpOkWyXAqdJHBPPYYTJ0KixaFnURkQCp0kYHE4/Dkk7BkiW5kIRlP31CRgTz3HOzbB5deGnYSkUGp0EUGsmpVsGV+0UVhJxEZlApdZCC/+U0wdz5xYthJRAalQhc5lt27obExmD8XyQIqdJFj+e1vg5+aP5csoUIXOZZVq6CmBs45J+wkIilRoYv0J5GAJ56ASy7R4YqSNfRNFenPCy/Anj2aP5esokIX6c+qVcF1Wy7WRUQle6jQRfrz1FPB3LlulShZRIUu0ld7OzzzDHzwg2EnERkSFbpIXw0NwdUVL7ww7CQiQ6JCF+mrrg7y82Hx4rCTiAyJCl2kr7o6eN/7oLw87CQiQ6JCF+mtrQ2ef17z55KVVOgivdXXB9dAV6FLFlKhi/RWVweFhXDeeWEnERkyFbpIb3V1sHAhlJaGnURkyFToIoe0tASn/Gu6RbKUCl3kkNWrg4tyqdAlS6nQRQ6pq4PiYnj/+8NOIjIsKnSRQ1avDm43V1ISdhKRYVGhiwAcOAAvvQTnnx92EpFhU6GLADz7bDB/rkKXLKZCF4HghKK8vGDKRSRLpVToZrbEzNab2QYzu/UYy1xgZi+b2Toz+0N6Y4qMsqefhve+Fyoqwk4iMmyDFrqZ5QP3AJcC84CrzGxen2WqgB8BH3b39wCfTH9UkVHS3R1MuejqipLlUtlCXwBscPeN7t4FPAws7bPMp4EV7r4FwN13pzemyCh68UXo6ND8uWS9VAp9GrC11+Ntyed6OxmYYGa/N7MXzOy6/lZkZjeaWaOZNTY1NQ0vsUi61dcHP7WFLlkulUK3fp7zPo8LgPnA5cAlwDfN7OSjfsn9XnevdffaGt2rUTLF00/DSSfB1KlhJxEZkVQKfRswo9fj6cCOfpZZ5e4H3X0PsBo4Kz0RRUaRe7CFrq1ziYBUCv15YK6ZzTKzIuBK4LE+y/wKON/MCsysFFgIvJ7eqCKj4I03YO9eFbpEQsFgC7h73MxuAZ4A8oEH3H2dmd2UfH2Zu79uZquAV4AEcJ+7vzqawUXS4tD8uXaISgQMWugA7r4SWNnnuWV9Ht8O3J6+aCJjoL4eJk+GuXPDTiIyYjpTVHJbfX1wdyLrb9+/SHZRoUvu2rkTNm7U7eYkMlTokrueeSb4ee654eYQSRMVuuSuhgYoKoJzzgk7iUhaqNAldzU0QG1tcJcikQhQoUtu6uyExkZNt0ikqNAlN734InR1aYeoRIoKXXJTQ0PwUzeElghRoUtuamiAOXNgypSwk4ikjQpdco87rFmj+XOJHBW65J5Nm2DXLhW6RI4KXXLPoflzFbpEjApdck9DQ3Az6Pe8J+wkImmlQpfc09AACxdCfn7YSUTSSoUuuaW1Fdau1fHnEkkqdMktzz0HiYTmzyWSVOiSWxoagmufL1wYdhKRtFOhS25paIAzzgh2iopEjApdckciEVwDXdMtElEqdMkdr70W7BRVoUtEqdAld+iEIok4FbrkjjVrYPJkmD077CQio0KFLrmjoSHYOjcLO4nIqFChS27YvRs2bNB0i0SaCl1ywzPPBD9V6BJhKnTJDQ0NUFgI8+eHnURk1KjQJTc0NARlXlISdhKRUaNCl+jr7ITnn9d0i0SeCl2i74UXglI///ywk4iMKhW6RN/TTwc/dclciTgVukRffT2ccgrU1ISdRGRUpVToZrbEzNab2QYzu3WA5d5nZj1m9on0RRQZgUQiOENU0y2SAwYtdDPLB+4BLgXmAVeZ2bxjLPd94Il0hxQZttdfh337YPHisJOIjLpUttAXABvcfaO7dwEPA0v7We4LwHJgdxrziYzMoflzFbrkgFQKfRqwtdfjbcnnDjOzacBHgWUDrcjMbjSzRjNrbGpqGmpWkaGrr4epU3VBLskJqRR6f1cy8j6P7wS+5u49A63I3e9191p3r63RDioZC/X1wfy5LsglOaAghWW2ATN6PZ4O7OizTC3wsAV/aaqBy8ws7u6PpiOkyLBs3QqbN8OXvxx2EpExkUqhPw/MNbNZwHbgSuDTvRdw91mHxmb2E+BxlbmErr4++Kn5c8kRgxa6u8fN7BaCo1fygQfcfZ2Z3ZR8fcB5c5HQ1NdDWRmceWbYSUTGRCpb6Lj7SmBln+f6LXJ3v37ksUTSoL4+uH5LQUpfc5GspzNFJZqam2HtWk23SE5RoUs0rV4N7nDBBWEnERkzKnSJpro6GDcOFiwIO4nImFGhSzTV1QVXVywuDjuJyJhRoUv0NDUF8+cf/GDYSUTGlApdomf16uCnCl1yjApdoqeuDsaPh9rasJOIjCkVukRPXV1wuGJhYdhJRMaUCl2iZdcueO01TbdITlKhS7T84Q/BTxW65CAVukRLXR2Ul8M554SdRGTMqdAlWurqguuf6/otkoNU6BId27fD+vWabpGcpUKX6HgieX/ySy4JN4dISFToEh2rVsHxx8Ppp4edRCQUKnSJhngcnnwSlizR/UMlZ6nQJRqeew727w8KXSRHqdAlGlatgrw8uOiisJOIhEaFLtGwahUsWgQTJoSdRCQ0KnTJfk1N0Nio6RbJeSp0yX5PPhncbk6FLjlOhS7Zb9UqqK6G+fPDTiISKhW6ZLdEIjih6OKLg52iIjlMfwMkuzU2wu7dmm4RQYUu2W758uBCXFdcEXYSkdCp0CV7ucOKFXDhhTpcUQQVumSztWthwwb4+MfDTiKSEVTokr2WLw+u27J0adhJRDKCCl2y14oVwc0spkwJO4lIRlChS3Z680149VVNt4j0klKhm9kSM1tvZhvM7NZ+Xr/azF5J/tdgZmelP6pILytWBD8/+tFwc4hkkEEL3czygXuAS4F5wFVmNq/PYpuAD7j7mcA/AvemO6jIuyxfDgsWwIwZYScRyRipbKEvADa4+0Z37wIeBt61F8rdG9x9X/Lhs8D09MYU6WXz5uCEIk23iLxLKoU+Ddja6/G25HPH8jngN/29YGY3mlmjmTU2NTWlnlKkt5/+NPj5yU+Gm0Mkw6RS6P3dz8v7XdDsgwSF/rX+Xnf3e9291t1ra2pqUk8pckgiAT/5SXAy0axZYacRySipFPo2oPdE5XRgR9+FzOxM4D5gqbvvTU88kT6efho2boQbbgg7iUjGSaXQnwfmmtksMysCrgQe672AmZ0ArACudfc30x9TJOnBB6GiAj72sbCTiGScgsEWcPe4md0CPAHkAw+4+zozuyn5+jLgW8Ak4EcW3HE97u61oxdbctKBA/CLX8A110BpadhpRDLOoIUO4O4rgZV9nlvWa/x54PPpjSbSxyOPQHu7pltEjkFnikr2ePBBOPVUWLgw7CQiGUmFLtlh/XpYswY++9ngglwichQVumSHO++EoiK47rqwk4hkLBW6ZL7du4Njz6+7TldWFBmACl0y3913Q2cnfOUrYScRyWgqdMlsBw/CPffAhz8Mp5wSdhqRjKZCl8z24IPQ3Ax/+7dhJxHJeCp0yVzxOPzgB/D+98N554WdRiTjpXRikUgoHn4Y3n4b7rgj7CQiWUFb6JKZOjrg61+Hs88O5s9FZFDaQpfMdMcdsHVrcO3zPG13iKRCf1Mk8+zcCd/9LnzkI3DBBWGnEckaKnTJPH/3dxCLwT//c9hJRLKKCl0yyyuvwH33wS23wNy5YacRySoqdMkcXV1w/fUwcSJ885thpxHJOtopKpnj29+Gl16CRx8NSl1EhkRb6JIZ6uvh+9+Hz30Oli4NO41IVlKhS/haW+Haa2HmTJ1EJDICmnKRcPX0BJfF3bIl2EovLw87kUjWUqFLeNzhr/8afvUruOuu4JotIjJsmnKR8Hz/+/CjH8FXvxocpigiI6JCl3Dcfz/cdhtcdVVwVqiIjJgKXcaWO3znO/D5z8PFFwfXO9e1WkTSQn+TZOzE4/BXfwXf+AZcfTX8+tdQXBx2KpHIUKHL2Ni+HS69FJYtg1tvDa6iWFQUdiqRSNFRLjL6Hnoo2DLv6oJ//ddgukVE0k5b6DJ63ngDPvpR+PSngxs8v/yyylxkFKnQJf22boW/+As4/XR46qlgJ2h9va6eKDLKNOUi6eEOdXXBceWPPhocuXLzzcEO0MmTw04nkhNU6DJ8iQT88Y/wy1/CihXw1lvBVRK//OVgznzmzLATiuQUFbqkrrMT1q2DNWtg9Wr4wx+gqQkKC+HCC+Fb34I//3MoKQk7qUhOSqnQzWwJ8EMgH7jP3b/X53VLvn4Z0A5c7+4vpjkrm5raqH9rL7taYuTngQHxBEypLGHmhHG8va/jqNcyYZxN+XY3H6S8ZQ8VTTsp27GFie9sYcK2t6neuJ6JWzeS3xMHoGXKNLafdS4b5y9my8IP0FVWEaz36c1j+mc5pbKExXMmMaum7Jjfj76/39/yvZ8XyVbm7gMvYJYPvAn8T2Ab8Dxwlbu/1muZy4AvEBT6QuCH7r5woPXW1tZ6Y2NjykE3NbXx88ZtVI0rJNYd59mNzThw7pyJHIz18NzmfSyYOYHSovzDr502tYw3draFOh61fFPG8+aOFiyR4D3VJby1Yz/5PXFOnVDE5nf2UxDv4uSKAnbs3EdhVydzxht7djZT0tnBiUVx2nbtpbS9jZn5nRQ1N8OeJqo7Winbt4e8RM/hP/eEGc1Vk2mefTLrambx9vS5sHABf+wpD/3P8oRJ42mLxdnf0c15syeyZmPzUd+Pvr9fXFBw1PJlJQWH1/Op2ukqdcloZvaCu9f2+1oKhf5+4Nvufkny8W0A7v7dXsv8GPi9uz+UfLweuMDd3znWeoda6P/27GYOxuKc8vIaTvs/3ybhwVaXWfB6T8LJzwseHHqtO5GgIC9vWON4Tw8FyVPS48nnAXp6EhTkB+8T70lQkHzPnp5eyyR6yE+ux3DMIeFOngEOuGN48DvJ/IlEggIcc3/X2BMJ8nHyEgmsp4c8T5CXSJDfq3SHI2F5tJeWExtfTlt5FfvLqohVTWT/hMnsrapm/4TJrBtXjc2eQ09RMdtb2jm+shSD0MZFhXng0NrRTUVpIYtmTQKCx6/uaOH04yupGFfIsxv30hlP9Pv7i2ZNOmr5Q1o7uhlfUsC1i04c0Z+tyGgaqNBTmXKZBmzt9XgbwVb4YMtMA95V6GZ2I3AjwAknnJDCWx+xqyXG1MoSusvK2XLcbArzg/Ls7kkAUJhvdPd4chy8tq+9iwmlRcMeV40Pxvvbu6kqLew1LgKDfQe7qBpffOR3k+Pm9i4mlBaDBfkco7Ag78g4Pw/M2NvexcTxxbgZe9q7mVhWAga72+NUl5XgBrsOxqmuKMHz8tjZ1k1NZSmJ/Dy2H+hm8oQyEvn5bD3QxdRJFfQUFLCppYvjplQRLyziv/d3MX1aNd1Fxaxt7mLmzKl0lpTy3N4uTp1zPOQZLR3dAFSUFNAaC6ZTKksKweDVLfs4u3QcBQYt7d3MnhTMXYQ2PpR1XAGtyTFAWUkB77R0sGh2suBj3Yc/Q3+/33f53uvZ2RIb0vdSJJOkUujWz3N9N+tTWQZ3vxe4F4It9BTe+7AplSW0xeLknTGfx798++EtsL5bbTj9bp1lylblWOfbmxzvqm4n/1CmRDuxHsd6PNhC7ZMvlsxXWVqYUeOjsia1xeIcVzmOtlicinGFVJQM8Pv9LN97PVMqtUNXslcqJxZtA2b0ejwd2DGMZUZk8ZxJ7O/oprWjm9nVpbS0d7GvvYs51aVMLS9mf0c3U8uL3/Xa/BlVoY+Vb/SyJtxp7egO5r7nT+/3+9H39/tbvvd6Fs+ZNPiXUSRDpTKHXkCwU/RDwHaCnaKfdvd1vZa5HLiFIztF/8XdFwy03qHOoYOOcsnVfDrKReSIEe0UTa7gMuBOgsMWH3D3/21mNwG4+7LkYYt3A0sIDlu8wd0HbOvhFLqISK4b6U5R3H0lsLLPc8t6jR24eSQhRURkZHRxLhGRiFChi4hEhApdRCQiVOgiIhGR0lEuo/LGZk3A5lDefGSqgT1hhxhj+szRl2ufF7L3M5/o7jX9vRBaoWcrM2s81iFDUaXPHH259nkhmp9ZUy4iIhGhQhcRiQgV+tDdG3aAEOgzR1+ufV6I4GfWHLqISERoC11EJCJU6CIiEaFCHwEz+4qZuZlVh51lNJnZ7Wb2hpm9Yma/NLOqsDONFjNbYmbrzWyDmd0adp7RZmYzzKzOzF43s3Vm9sWwM40VM8s3s5fM7PGws6SLCn2YzGwGwY2zt4SdZQw8CZzu7mcSXBv/tpDzjIrkDdHvAS4F5gFXmdm8cFONujjwv9z9NGARcHMOfOZDvgi8HnaIdFKhD98dwFfp51Z7UePuv3X3ePLhswR3pIqiBcAGd9/o7l3Aw8DSkDONKnd/x91fTI4PEBTctHBTjT4zmw5cDtwXdpZ0UqEPg5l9GNju7n8KO0sIPgv8JuwQo+RYNzvPCWY2Ezgb+GPIUcbCnQQbZImQc6RVSje4yEVm9hQwtZ+XvgF8Hbh4bBONroE+r7v/KrnMNwj+if7vY5ltDKV0s/MoMrMyYDnwJXdvDTvPaDKzK4Dd7v6CmV0Qcpy0UqEfg7tf1N/zZnYGMAv4U3DnPaYDL5rZAnffOYYR0+pYn/cQM/sMcAXwIY/uyQujfrPzTGRmhQRl/u/uviLsPGPgPODDyVtrlgAVZvYzd78m5FwjphOLRsjM3gZq3T0br9qWEjNbAvxf4APu3hR2ntGSyg3RoyZ5P+D/BzS7+5dCjjPmklvoX3H3K0KOkhaaQ5dU3A2UA0+a2ctmtmywX8hGyR2/twBPEOwcfCTKZZ50HnAtcGHy/+3LyS1XyULaQhcRiQhtoYuIRIQKXUQkIlToIiIRoUIXEYkIFbqISESo0EVEIkKFLiISEf8f2LgPZ0EiF74AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "max_iter = 5000\n",
    "i = 0\n",
    "descent = numpy.ones(len(params))\n",
    "\n",
    "while numpy.linalg.norm(descent) > 0.001 and i < max_iter:\n",
    "\n",
    "    descent = gradient(params, logistic_model, x_data, y_data)\n",
    "    params = params - descent * 0.01\n",
    "    i += 1\n",
    "\n",
    "\n",
    "print(f'Optimized value of w is {params[0]:.3f} vs. true value: 2')\n",
    "print(f'Optimized value of b is {params[1]:.3f} vs. true value: 1')\n",
    "print(f'Exited after {i} iterations')\n",
    "\n",
    "\n",
    "pyplot.scatter(x_data, y_data, alpha=0.4)\n",
    "pyplot.plot(x_data, logistic_model(params, x_data), '-r');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It worked! We found a logistic curve that approximates the data quite well. \n",
    "This curve represents the _prediction model_ for the continuous data between $0$ and $1$. \n",
    "In a classification task, the next step is to assign a decision threshold for the two classes; a natural choice is $0.5$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decision_boundary(y):\n",
    "    return 1 if y >= .5 else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function will code as $0$ or $1$ each data point. We will want to apply the function element-wise to all elements in an array, for which we can use [`numpy.vectorize()`](https://numpy.org/doc/stable/reference/generated/numpy.vectorize.html). \n",
    "Be sure to read the documentation of this very useful NumPy function that saves you from having to write a for-loop!\n",
    "\n",
    "The decision rule is what turns logistic regression into a classification tool. Check it out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "decision_boundary = numpy.vectorize(decision_boundary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAafUlEQVR4nO3df3BV5b3v8fc3O4EdCSQVUlCgJIMcASVAjSAFrV78gcLg2NFD9ehR7h2ttVhPvUeL196WsZ253lNbj2fUIsc62Nai1eOtP8ajHI/SK1W5Booo8sMgAhEMMZiQlPzYSb73jx1iCDvJTrLDTh4/rxkma63n2Wt/V9z99Mmz1l7L3B0RERn8MtJdgIiIpIYCXUQkEAp0EZFAKNBFRAKhQBcRCURmut541KhRXlBQkK63FxEZlDZu3PiZu+cnaktboBcUFFBSUpKutxcRGZTMbE9nbZpyEREJhAJdRCQQCnQRkUCkbQ49kVgsRllZGfX19ekuJSjRaJRx48aRlZWV7lJEpB8NqEAvKytj+PDhFBQUYGbpLicI7k5lZSVlZWUUFhamuxwR6UfdBrqZPQYsAg66+5kJ2g14ALgMOALc4O6belNMfX29wjzFzIyRI0dSUVGR7lL6xe6KWtbvqqS8up7RuVHmTRxJYX5Op/33fbiFT/7yMl69n2h0KBNOPomToxlU1rWw79AR6hoacIsAYN7c42XLPZWT8k/jSEUpXr2/T/vq7+WBXutAr6+vtVruqYyduYDxk4pIlWRG6KuBB4HfdNJ+KTCp9d9s4FetP3tFYZ56of5Od1fU8lRJGXnZWYzJjVJb38RTJWUsKR6XMNT3fbiFstf+FbLziEazyf9sA4cPOkdGn0lT+fuMyjDKoxOJHtpGC1Az/HRya3YkvzxmDv7ZbobseIbaUcV41jDyyt/q3b76e3mg1zrQ6+trrWPmQH1N/PPIjSkLdUvm9rlmVgC82MkI/RFgnbuvaV3fAZzv7ge62mdxcbF3vA5927ZtTJkyJfnqJWkh/m5/+/Ye/lrfxIjsL84NHK6LMSyayXXnTDiu/9t/+Ce8voaM7Dy+WllCRksDTS1OpOYTmnNOJTOSgVeXURn5KgYMazhA7dBTkl5ujgwBICt2mFjWCAAymht7ta/+Xh7otQ70+vpaa3NkCC1fm0tLXRUWHc45f3tnch96wMw2untxorZUXOUyFtjXbr2sdVuiQm4ysxIzKxmIUwBVVVU8/PDD6S6jzerVq1m2bFm6yxiwyqvryYke+0dmTjST8urEJ9W9ej8WzQUgq6mG5oyhWFY2Q2LVWFY2zZEo0ebDtGQMpTkylJOaa3u0nNVUQ1ashljm8Lbl3u6rv5cHeq0Dvb6+1prVVAOARXPx6i7Hvj2SipOiif6eTzjsd/dVwCqIj9BT8N4pdTTQb7nlluPampubiUQiaahKOjO6dZql/Qi9tr6J0bnRhP0t91S8vhrLziOWOZxI6wi9MSsXj9WRGcmgPjKCjJYGDDgSyenRcixzONBxpNa7ffX38kCvdaDX19daj77G66ux3FO6+aQnLxUj9DJgfLv1ccD+FOy3W7sravnt23u475Ud/PbtPeyuqO3T/pYvX86uXbuYMWMGd9xxB+vWreOCCy7gmmuuYdq0aXz88ceceeYXs0733XcfK1asAGDXrl0sWLCAs846i3PPPZft27cfs++WlhYKCgqoqqpq23baaadRXl7OCy+8wOzZs5k5cyYXXngh5eXlx9V2ww038Mwzz7St5+R8MUf885//nLPPPpuioiJ+8pOf9Ol3MJjMmziSqroYh+titLhzuC5GVV2MeRNHJuw/duYCqKuipa6KqmEFRBqqiDRUkTl+NpHGaiINVVTmTScaqyIrVkVl7oweLcfyJhLLHk00dpjYSWNozJ3Y63319/JAr3Wg19fXWmN5E2mpq4K6qvjnMkVSMYe+EFhG/CqX2cC/uPus7vbZ1zn09ifEcqKZ1NY3UVUX6/SEWDI+/vhjFi1axPvvvw/AunXrWLhwIe+//z6FhYXHtd93333U1tayYsUK5s+fz8qVK5k0aRIbNmzgrrvu4rXXXjtm/7fddhszZsxg6dKlbNiwgbvvvptXX32Vzz//nLy8PMyMRx99lG3btvGLX/yC1atXU1JSwoMPPsgNN9zAokWLuPLKK4F4oNfW1rJ27VqeeeYZHnnkEdydxYsXc+edd3Leeef1+nc7mPT+KpcDRKND4le5ZEeoPNLcepVLI27xcY55S4+XLfeUdlc7HOjTvvp7eaDXOtDr62utlntKr65y6WoOPZnLFtcA5wOjzKwM+AmQBeDuK4GXiId5KfHLFpf2qLpeWr+rkrzsrLY/t4/+XL+rsteBnsisWbO6vX67traWN998k6uuuqptW0NDw3H9lixZwj333MPSpUt58sknWbJkCRC//n7JkiUcOHCAxsbGHl0vvnbtWtauXcvMmTPbavnwww+PC/RQFebn9Oi/9/hJRQn/BzSy9Z/IYNZtoLv71d20O/C9lFWUpPLqesZ0mCvNiWbyaScnxHpr2LBhbcuZmZm0tLS0rR/9RmtLSwt5eXls3ry5y33NmTOH0tJSKioq+OMf/8iPfvQjAG699VZuv/12Fi9ezLp169qmcdpr/97uTmNjY9vyXXfdxXe+852+HKaIBGDQ3svl6Amx9ro6IZaM4cOHU1NT0/l7jh7NwYMHqayspKGhgRdffBGAESNGUFhYyNNPPw3EQ/bdd9897vVmxhVXXMHtt9/OlClTGDkyPiasrq5m7Nj4hUGPP/54wvcuKChg48aNADz33HPEYjEALrnkEh577DFqa+PnDz755BMOHjzYm8MXkUFu0AZ6T0+IJWPkyJHMnTuXM888kzvuuOO49qysLH784x8ze/ZsFi1axOTJk9vannjiCX79618zffp0zjjjDJ577rmE77FkyRJ+97vftU23AKxYsYKrrrqKc889l1GjRiV83Y033sif/vQnZs2axYYNG9r+crj44ou55pprmDNnDtOmTePKK6/s8v+URCRcSZ0U7Q+p+GJRT0+IfZmFelJU5MumTydFB7KenhATEQnZoJ1yERGRYynQRUQCoUAXEQmEAl1EJBAKdBGRQCjQ+9G6detYtGgRAM8//zz33ntvl/2/8Y1vAPF7yvz+97/v9/pEJCwK9F5obm7u8WsWL17M8uXLu+zz5ptvAgp0EemdwR3olbvg/z0K//nT+M/KXX3e5ccff8zkyZO5/vrrKSoq4sorr+TIkSMUFBRwzz33MG/ePJ5++mnWrl3LnDlz+PrXv85VV13V9tX7l19+mcmTJzNv3jyeffbZtv22f1hFeXk5V1xxBdOnT2f69OltQX70lrjLly/njTfeYMaMGdx///3U19ezdOlSpk2bxsyZM3n99dfb9vmtb32LBQsWMGnSJO68M/mnnohIeAZvoFfugk2/gcZaGHFq/Oem36Qk1Hfs2MFNN93Eli1bGDFiRNtTjKLRKOvXr+fCCy/kZz/7Ga+++iqbNm2iuLiYX/7yl9TX13PjjTfywgsv8MYbb/Dpp58m3P/3v/99vvnNb/Luu++yadMmzjjjjGPa7733Xs4991w2b97MD37wAx566CEA3nvvPdasWcP111/fdmOwzZs389RTT/Hee+/x1FNPsW/fvuPeT0S+HAZvoO96HbK/AtFcsIz4z+yvxLf30fjx45k7dy4A1157LevXrwdou//K22+/zQcffMDcuXOZMWMGjz/+OHv27GH79u0UFhYyadIkzIxrr7024f5fe+01vvvd7wIQiUTIzc3tsp7169dz3XXXATB58mQmTJjAzp07AZg/fz65ublEo1GmTp3Knj17+nz8IjI4Dd6v/tcciI/M2xs6PL69j8ws4frRG2K5OxdddBFr1qw5pt/mzZuPe20qdHW/naFDh7YtRyIRmpqaOu0rImEbvCP04adAQ4e7CjbUQM6YPu967969vPXWWwCsWbOGefPmHdN+zjnn8Oc//5nS0lIAjhw5ws6dO5k8eTK7d+9m165dba9NZP78+fzqV78C4idYDx8+fOyhdbiN73nnnccTTzwBwM6dO9m7dy+nn356n49TRMIyeAN94gVQ9znUV4O3xH/WfR7f3kdTpkzh8ccfp6ioiEOHDrVNjxyVn5/P6tWrufrqqykqKuKcc85h+/btRKNRVq1axcKFC5k3bx4TJkxIuP8HHniA119/nWnTpnHWWWexdevWY9qLiorIzMxk+vTp3H///dxyyy00Nzczbdo0lixZwurVq48ZmYuIwCC/fS6Vu+Jz5rWfxkfmEy+AkRP7VFfH54aGQrfPFQlDsLfPZeTEPge4iEgoBu+USz8pKCgIbnQuIl8OAy7Q0zUFFDL9TkW+HAZUoEejUSorKxVAKeTuVFZWEo32/uHZIjI4DKg59HHjxlFWVkZFRUW6SwlKNBpl3Lhx6S5DRPrZgAr0rKwsCgsL012GiMigNKCmXEREpPcU6CIigVCgi4gEQoEuIhIIBbqISCAU6CIigVCgi4gEIqlAN7MFZrbDzErN7LgnHZtZrpm9YGbvmtlWM1ua+lJFRKQr3Qa6mUWAh4BLganA1WY2tUO37wEfuPt04HzgF2Y2JMW1iohIF5IZoc8CSt39I3dvBJ4ELu/Qx4HhFn/+Wg5wCNCz0ERETqBkAn0s0P5R8mWt29p7EJgC7AfeA25z95aOOzKzm8ysxMxKdL8WEZHUSibQEz31uOPtEC8BNgOnAjOAB81sxHEvcl/l7sXuXpyfn9/DUkVEpCvJBHoZML7d+jjiI/H2lgLPelwpsBuYnJoSRUQkGckE+jvAJDMrbD3R+W3g+Q599gLzAcxsNHA68FEqCxURka51e/tcd28ys2XAK0AEeMzdt5rZza3tK4GfAqvN7D3iUzQ/dPfP+rFuERHpIKn7obv7S8BLHbatbLe8H7g4taWJiEhP6JuiIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiAQiqUA3swVmtsPMSs1seSd9zjezzWa21cz+lNoyRUSkO5nddTCzCPAQcBFQBrxjZs+7+wft+uQBDwML3H2vmX21n+oVEZFOJDNCnwWUuvtH7t4IPAlc3qHPNcCz7r4XwN0PprZMERHpTjKBPhbY1269rHVbe38DfMXM1pnZRjP7+0Q7MrObzKzEzEoqKip6V7GIiCSUTKBbgm3eYT0TOAtYCFwC/E8z+5vjXuS+yt2L3b04Pz+/x8WKiEjnup1DJz4iH99ufRywP0Gfz9z9r8Bfzez/AtOBnSmpUkREupXMCP0dYJKZFZrZEODbwPMd+jwHnGtmmWZ2EjAb2JbaUkVEpCvdjtDdvcnMlgGvABHgMXffamY3t7avdPdtZvYysAVoAR519/f7s3ARETmWuXecDj8xiouLvaSkJC3vLSIyWJnZRncvTtSmb4qKiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiARCgS4iEggFuohIIJIKdDNbYGY7zKzUzJZ30e9sM2s2sytTV6KIiCSj20A3swjwEHApMBW42symdtLvfwOvpLpIERHpXjIj9FlAqbt/5O6NwJPA5Qn63Qr8G3AwhfWJiEiSkgn0scC+dutlrdvamNlY4ApgZVc7MrObzKzEzEoqKip6WquIiHQhmUC3BNu8w/o/Az909+auduTuq9y92N2L8/PzkyxRRESSkZlEnzJgfLv1ccD+Dn2KgSfNDGAUcJmZNbn7H1NRpIiIdC+ZQH8HmGRmhcAnwLeBa9p3cPfCo8tmthp4UWEuInJidRvo7t5kZsuIX70SAR5z961mdnNre5fz5iIicmIkM0LH3V8CXuqwLWGQu/sNfS9LRER6St8UFREJhAJdRCQQCnQRkUAo0EVEAqFAFxEJhAJdRCQQCnQRkUAo0EVEAqFAFxEJhAJdRCQQCnQRkUAo0EVEAqFAFxEJhAJdRCQQCnQRkUAo0EVEAqFAFxEJhAJdRCQQCnQRkUAo0EVEAqFAFxEJhAJdRCQQCnQRkUAo0EVEAqFAFxEJhAJdRCQQCnQRkUAo0EVEAqFAFxEJhAJdRCQQSQW6mS0wsx1mVmpmyxO0/52ZbWn996aZTU99qSIi0pVuA93MIsBDwKXAVOBqM5vaodtu4JvuXgT8FFiV6kJFRKRryYzQZwGl7v6RuzcCTwKXt+/g7m+6++etq28D41JbpoiIdCeZQB8L7Gu3Xta6rTP/Dfj3RA1mdpOZlZhZSUVFRfJViohIt5IJdEuwzRN2NLuAeKD/MFG7u69y92J3L87Pz0++ShER6VZmEn3KgPHt1scB+zt2MrMi4FHgUnevTE15IiKSrGRG6O8Ak8ys0MyGAN8Gnm/fwcy+BjwLXOfuO1NfpoiIdKfbEbq7N5nZMuAVIAI85u5bzezm1vaVwI+BkcDDZgbQ5O7F/Ve2iIh0ZO4Jp8P7XXFxsZeUlKTlvUVEBisz29jZgFnfFBURCYQCXUQkEAp0EZFAKNBFRAKhQBcRCYQCXUQkEAp0EZFAKNBFRAKhQBcRCYQCXUQkEAp0EZFAKNBFRAKhQBcRCYQCXUQkEAp0EZFAKNBFRAKhQBcRCYQCXUQkEAp0EZFAKNBFRAKhQBcRCYQCXUQkEAp0EZFAKNBFRAKhQBcRCYQCXUQkEAp0EZFAKNBFRAKhQBcRCYQCXUQkEAp0EZFAZCbTycwWAA8AEeBRd7+3Q7u1tl8GHAFucPdNKa6VfR9u4ZO/vIxX78ctEn9vb8ZyT+Wk/NM4UlF6XNtAWFZ9/VdrNDqUCSefxMnRDCrrWth36Ah1DQ1dvn7szAWMn1TE7opa1u+qpLy6ntG5UeZNHElhfg4ig5W5e9cdzCLATuAioAx4B7ja3T9o1+cy4FbigT4beMDdZ3e13+LiYi8pKUm60H0fbqHstX+F7DxaGuvJK3+LFqBmzBxoqOXkyo0cGlWMZw37om346eTW7Ejvsurrt1qzc3LJ/2wDTS1O5ugzaSp/n8wMozw6keihbQlfb1lRqKsi8+vX8Z+fjSAvO4ucaCa19U1U1cVYUjxOoS4DmpltdPfihG1JBPocYIW7X9K6fheAu/+vdn0eAda5+5rW9R3A+e5+oLP99jTQ3/7DP+H1NWRk55Gx989kNDdiQHNkCABZscPEskYAtLUNazhA7dBT0rqs+vqv1lHDhpLR0kBTixOp+YTmnFPJjGTg1WVURr6a8PUtX5tLS10Ve2ozqJr694zIzmr7jB2uizEsmsl150xI+nMpcqJ1FejJzKGPBfa1Wy9r3dbTPpjZTWZWYmYlFRUVSbz1F7x6PxbNBSArVkNLxlCaI0PJaqohK1ZDLHN42/LRtpOaa9O+rPr6sdamGpozhmJZ2QyJVWNZ2TRHokSbD3f6egCL5hKp/ZSc6LEzjjnRTMqr63v0uRQZSJKZQ7cE2zoO65Ppg7uvAlZBfISexHt/8Qa5p+L11Vh2HrGs4WQ0N2BALHM40HGEGW87EskhoyW9y6qv/2qNZQ4l0jpCb8zKxWN1ZEYyqI+M6PL1Xl9Nc84Yauubjhmh19Y3MTo32pOPpciAkswIvQwY3259HLC/F336ZOzMBVBXRUtdFY25E4nGqsiKVRHLm0gsezTR2GFiJ405pq0yd0bal1Vf/9VaNayASEMVkYYqMsfPJtJYTaShisq86Z2+vqWuCuqqmHj2QqrqYhyui9HizuG6GFV1MeZNHJnKj63ICZXMHHom8ZOi84FPiJ8Uvcbdt7brsxBYxhcnRf/F3Wd1td+ezqFD+6tcDuCW0XoALVjuKe2ufDi2bSAsq77+qzUaHRK/yiU7QuWR5tarXBq7fL2ucpHBrKs59G6nXNy9ycyWAa8Qv2zxMXffamY3t7avBF4iHualxC9bXJqq4tsbP6mI8ZOK+mPXEoCRrf+SVZifowCXoCR1Hbq7v0Q8tNtvW9lu2YHvpbY0ERHpCX1TVEQkEAp0EZFAKNBFRAKhQBcRCUS3ly322xubVQB70vLmfTMK+CzdRZxgOubwfdmOFwbvMU9w9/xEDWkL9MHKzEo6uwY0VDrm8H3ZjhfCPGZNuYiIBEKBLiISCAV6z61KdwFpoGMO35fteCHAY9YcuohIIDRCFxEJhAJdRCQQCvQ+MLN/NDM3s1HprqU/mdnPzWy7mW0xs/9jZnnprqm/mNkCM9thZqVmtjzd9fQ3MxtvZq+b2TYz22pmt6W7phPFzCJm9hczezHdtaSKAr2XzGw88Qdn7013LSfAfwBnunsR8Xvj35XmevpF6wPRHwIuBaYCV5vZ1PRW1e+agP/u7lOAc4DvfQmO+ajbgG3pLiKVFOi9dz9wJwketRcad1/r7k2tq28TfyJViGYBpe7+kbs3Ak8Cl6e5pn7l7gfcfVPrcg3xgDvuecChMbNxwELg0XTXkkoK9F4ws8XAJ+7+brprSYP/Cvx7uovoJ0k97DxUZlYAzAQ2pLmUE+GfiQ/IWtJcR0ol9YCLLyMzexUYk6DpbuB/ABef2Ir6V1fH6+7Ptfa5m/if6E+cyNpOoKQedh4iM8sB/g34B3c/nO56+pOZLQIOuvtGMzs/zeWklAK9E+5+YaLtZjYNKATeNTOITz9sMrNZ7v7pCSwxpTo73qPM7HpgETDfw/3yQr8/7HwgMrMs4mH+hLs/m+56ToC5wGIzuwyIAiPM7Hfufm2a6+ozfbGoj8zsY6DY3QfjXduSYmYLgF8C33T3inTX01+SeSB6aCw+KnkcOOTu/5Dmck641hH6P7r7ojSXkhKaQ5dkPAgMB/7DzDab2cruXjAYtZ74PfpA9G3AH0IO81ZzgeuA/9L633Zz68hVBiGN0EVEAqERuohIIBToIiKBUKCLiARCgS4iEggFuohIIBToIiKBUKCLiATi/wPrGIIwZqpdRQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def classify(predictions):\n",
    "    '''\n",
    "    Argument:\n",
    "    predictions, an array of values between 0 and 1\n",
    "    \n",
    "    Returns: \n",
    "    classified, an array of 0 and 1 values'''\n",
    "\n",
    "    return decision_boundary(predictions).flatten()\n",
    "\n",
    "pyplot.scatter(x_data, y_data, alpha=0.4,\n",
    "               label='true value')\n",
    "pyplot.scatter(x_data, classify(logistic_model(params, x_data)), alpha=0.4, \n",
    "               label='prediciton')\n",
    "\n",
    "pyplot.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this is the end of Lesson 2 of our module. We spent some time deriving the logistic loss function, where most sources will just give it to you without explanation. \n",
    "We find it more satisfying this way, but you can certainly skip it. \n",
    "\n",
    "The key ingredients we have added in this lesson are the ideas of:\n",
    "- using composition of functions in our model\n",
    "- using automatic differentiation to get the gradients for the optimization loop\n",
    "\n",
    "In the next lesson, we add the idea of multiple independent variables (a.k.a. _features_) affecting our model, leading to **multiple linear regression.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What we've learned\n",
    "\n",
    "- The logistic function approximates a step function while being differentiable. That's nice.\n",
    "- Logistic regression applies composition of the logistic function with a linear model.\n",
    "- We can use logistic regression in binary classification problems.\n",
    "- Using the square-error in this case leads to a non-convex loss function, with local minima.\n",
    "- The logistic loss function is best here for optimizing with gradient descent.\n",
    "- Automatic differentiation gives us derivatives of numeric functions written in computer code.\n",
    "- We pick some values in the optimization loop that we need to be careful with (e.g., learning rate)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "1. Eric Ma, \"Deep Learning Fundamentals: Forward Model, Differentiable Loss Function & Optimization,\" SciPy 2019 tutorial. [video on YouTube](https://youtu.be/JPBz7-UCqRo) and [archive on GitHub](https://github.com/ericmjl/dl-workshop/releases/tag/scipy2019).\n",
    "2. Michael A. Nielsen, \"Neural Networks and Deep Learning\" (2015), Determination Press, http://neuralnetworksanddeeplearning.com"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link href=\"https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i\" rel='stylesheet' >\n",
       "<link href=\"https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i\" rel='stylesheet' >\n",
       "<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >\n",
       "<style>\n",
       "\n",
       "@font-face {\n",
       "    font-family: \"Computer Modern\";\n",
       "    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "}\n",
       "\n",
       "\n",
       "#notebook_panel { /* main background */\n",
       "    background: rgb(245,245,245);\n",
       "}\n",
       "\n",
       "div.cell { /* set cell width */\n",
       "    width: 800px;\n",
       "}\n",
       "\n",
       "div #notebook { /* centre the content */\n",
       "    background: #fff; /* white background for content */\n",
       "    width: 1000px;\n",
       "    margin: auto;\n",
       "    padding-left: 0em;\n",
       "}\n",
       "\n",
       "#notebook li { /* More space between bullet points */\n",
       "margin-top:0.5em;\n",
       "}\n",
       "\n",
       "/* draw border around running cells */\n",
       "div.cell.border-box-sizing.code_cell.running { \n",
       "    border: 1px solid #111;\n",
       "}\n",
       "\n",
       "/* Put a solid color box around each cell and its output, visually linking them*/\n",
       "div.cell.code_cell {\n",
       "    background-color: rgb(256,256,256); \n",
       "    border-radius: 0px; \n",
       "    padding: 0.5em;\n",
       "    margin-left:1em;\n",
       "    margin-top: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render{\n",
       "    font-family: 'Source Sans Pro', sans-serif;\n",
       "    line-height: 140%;\n",
       "    font-size: 110%;\n",
       "    width:680px;\n",
       "    margin-left:auto;\n",
       "    margin-right:auto;\n",
       "}\n",
       "\n",
       "/* Formatting for header cells */\n",
       ".text_cell_render h1 {\n",
       "    font-family: 'Merriweather', serif;\n",
       "    font-style:regular;\n",
       "    font-weight: bold;    \n",
       "    font-size: 250%;\n",
       "    line-height: 100%;\n",
       "    color: #004065;\n",
       "    margin-bottom: 1em;\n",
       "    margin-top: 0.5em;\n",
       "    display: block;\n",
       "}\t\n",
       ".text_cell_render h2 {\n",
       "    font-family: 'Merriweather', serif;\n",
       "    font-weight: bold; \n",
       "    font-size: 180%;\n",
       "    line-height: 100%;\n",
       "    color: #0096d6;\n",
       "    margin-bottom: 0.5em;\n",
       "    margin-top: 0.5em;\n",
       "    display: block;\n",
       "}\t\n",
       "\n",
       ".text_cell_render h3 {\n",
       "    font-family: 'Merriweather', serif;\n",
       "\tfont-size: 150%;\n",
       "    margin-top:12px;\n",
       "    margin-bottom: 3px;\n",
       "    font-style: regular;\n",
       "    color: #008367;\n",
       "}\n",
       "\n",
       ".text_cell_render h4 {    /*Use this for captions*/\n",
       "    font-family: 'Merriweather', serif;\n",
       "    font-weight: 300; \n",
       "    font-size: 100%;\n",
       "    line-height: 120%;\n",
       "    text-align: left;\n",
       "    width:500px;\n",
       "    margin-top: 1em;\n",
       "    margin-bottom: 2em;\n",
       "    margin-left: 80pt;\n",
       "    font-style: regular;\n",
       "}\n",
       "\n",
       ".text_cell_render h5 {  /*Use this for small titles*/\n",
       "    font-family: 'Source Sans Pro', sans-serif;\n",
       "    font-weight: regular;\n",
       "    font-size: 130%;\n",
       "    color: #e31937;\n",
       "    font-style: italic;\n",
       "    margin-bottom: .5em;\n",
       "    margin-top: 1em;\n",
       "    display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h6 { /*use this for copyright note*/\n",
       "    font-family: 'Source Code Pro', sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 9pt;\n",
       "    line-height: 100%;\n",
       "    color: grey;\n",
       "    margin-bottom: 1px;\n",
       "    margin-top: 1px;\n",
       "}\n",
       "\n",
       "    .CodeMirror{\n",
       "            font-family: \"Source Code Pro\";\n",
       "\t\t\tfont-size: 90%;\n",
       "    }\n",
       "/*    .prompt{\n",
       "        display: None;\n",
       "    }*/\n",
       "\t\n",
       "    \n",
       "    .warning{\n",
       "        color: rgb( 240, 20, 20 )\n",
       "        }  \n",
       "</style>\n",
       "<script>\n",
       "    MathJax.Hub.Config({\n",
       "                        TeX: {\n",
       "                           extensions: [\"AMSmath.js\"], \n",
       "                           equationNumbers: { autoNumber: \"AMS\", useLabelIds: true}\n",
       "                           },\n",
       "                tex2jax: {\n",
       "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "                },\n",
       "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "                \"HTML-CSS\": {\n",
       "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "                }\n",
       "        });\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Execute this cell to load the notebook's style sheet, then ignore it\n",
    "from IPython.core.display import HTML\n",
    "css_file = '../style/custom.css'\n",
    "HTML(open(css_file, \"r\").read())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}