GA Challenge Part I.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Coding and Data Set\n",
    "Load in the data file and header file provided\n",
    "The dataframe does not currently have a header, load in the header file and attach it to the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>radius_sd_error</th>\n",
       "      <th>radius_worst</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>texture_sd_error</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>perimeter_sd_error</th>\n",
       "      <th>...</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave_points_mean</th>\n",
       "      <th>concave_points_sd_error</th>\n",
       "      <th>concave_points_worst</th>\n",
       "      <th>symmetry_mean</th>\n",
       "      <th>symmetry_sd_error</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_mean</th>\n",
       "      <th>fractal_dimension_sd_error</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         ID diagnosis  radius_mean  radius_sd_error  radius_worst  \\\n",
       "0    842302         M        17.99            10.38        122.80   \n",
       "1    842517         M        20.57            17.77        132.90   \n",
       "2  84300903         M        19.69            21.25        130.00   \n",
       "3  84348301         M        11.42            20.38         77.58   \n",
       "4  84358402         M        20.29            14.34        135.10   \n",
       "\n",
       "   texture_mean  texture_sd_error  texture_worst  perimeter_mean  \\\n",
       "0        1001.0           0.11840        0.27760          0.3001   \n",
       "1        1326.0           0.08474        0.07864          0.0869   \n",
       "2        1203.0           0.10960        0.15990          0.1974   \n",
       "3         386.1           0.14250        0.28390          0.2414   \n",
       "4        1297.0           0.10030        0.13280          0.1980   \n",
       "\n",
       "   perimeter_sd_error  ...  concavity_worst  concave_points_mean  \\\n",
       "0             0.14710  ...            25.38                17.33   \n",
       "1             0.07017  ...            24.99                23.41   \n",
       "2             0.12790  ...            23.57                25.53   \n",
       "3             0.10520  ...            14.91                26.50   \n",
       "4             0.10430  ...            22.54                16.67   \n",
       "\n",
       "   concave_points_sd_error  concave_points_worst  symmetry_mean  \\\n",
       "0                   184.60                2019.0         0.1622   \n",
       "1                   158.80                1956.0         0.1238   \n",
       "2                   152.50                1709.0         0.1444   \n",
       "3                    98.87                 567.7         0.2098   \n",
       "4                   152.20                1575.0         0.1374   \n",
       "\n",
       "   symmetry_sd_error  symmetry_worst  fractal_dimension_mean  \\\n",
       "0             0.6656          0.7119                  0.2654   \n",
       "1             0.1866          0.2416                  0.1860   \n",
       "2             0.4245          0.4504                  0.2430   \n",
       "3             0.8663          0.6869                  0.2575   \n",
       "4             0.2050          0.4000                  0.1625   \n",
       "\n",
       "   fractal_dimension_sd_error  fractal_dimension_worst  \n",
       "0                      0.4601                  0.11890  \n",
       "1                      0.2750                  0.08902  \n",
       "2                      0.3613                  0.08758  \n",
       "3                      0.6638                  0.17300  \n",
       "4                      0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "file = open('field_names.txt', 'r')\n",
    "field_names = file.read().split('\\n')\n",
    "df = pd.read_csv('breast-cancer.csv',names=field_names)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comment on any steps you might take to evaluate or transform the dataset.\n",
    "##### Evaluate the dataset\n",
    "Looking at the data, I notice that the column names don't correctly match the data. You can see this because the worst values are frequently far different then the mean values for the same field. For example, the first row has texture_mean = 1001.0, but texture_worst = 0.27760. It doesn't make sense that the worst value is so much _smaller_ than the mean, particularly given the low standard error. If I were dealing with this problem in the real world, I'd start by confirming that the column headings are really correctly ordered relatively to the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compute the mean and median smoothness and compactness for benign and malignant tumors - do they differ?\n",
    "Yes, the values for median and mean smoothness are higher for malignant tumors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>diagnosis</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>B</td>\n",
       "      <td>1.8510</td>\n",
       "      <td>0.01631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>M</td>\n",
       "      <td>3.6795</td>\n",
       "      <td>0.02859</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           smoothness_mean  compactness_mean\n",
       "diagnosis                                   \n",
       "B                   1.8510           0.01631\n",
       "M                   3.6795           0.02859"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#median smoothness\n",
    "df.groupby(['diagnosis']).median()[['smoothness_mean','compactness_mean']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>diagnosis</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>B</td>\n",
       "      <td>2.000321</td>\n",
       "      <td>0.021438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>M</td>\n",
       "      <td>4.323929</td>\n",
       "      <td>0.032281</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           smoothness_mean  compactness_mean\n",
       "diagnosis                                   \n",
       "B                 2.000321          0.021438\n",
       "M                 4.323929          0.032281"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#mean smoothness\n",
    "df.groupby(['diagnosis']).mean()[['smoothness_mean','compactness_mean']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1d534a4be88>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#### Box plots for distribution of mean and median smoothness and compactness\n",
    "import matplotlib.pyplot as plt\n",
    "fig, ax = plt.subplots(1, 3, figsize=(10, 5))\n",
    "df.boxplot('smoothness_mean','diagnosis', ax=ax[0])\n",
    "df.boxplot('compactness_mean','diagnosis', ax=ax[1])\n",
    "df.boxplot('radius_mean','diagnosis', ax=ax[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write a function to generate bootstrap samples of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import resample\n",
    "def bootstrap_samples(dataframe,number_of_samples,sample_size):\n",
    "    samples = []\n",
    "    for x in range(0, number_of_samples):\n",
    "        samples.append(resample(df, replace=True, n_samples=number_of_samples, random_state=1))    \n",
    "    return samples\n",
    "\n",
    "samples_array = bootstrap_samples(df,len(df),30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory Analysis\n",
    "Identify 2-3 variables that are predictive of a malignant tumor.\n",
    "Display the relationship visually and write 1-2 sentences explaining the relationship"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "diagnosisInt               1.000000\n",
       "fractal_dimension_mean     0.793566\n",
       "concave_points_sd_error    0.782914\n",
       "perimeter_sd_error         0.776614\n",
       "Name: diagnosisInt, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['diagnosisInt']=(df['diagnosis']=='B').astype(int)\n",
    "df.corr()['diagnosisInt'].abs().nlargest(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>radius_sd_error</th>\n",
       "      <th>radius_worst</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>texture_sd_error</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>perimeter_sd_error</th>\n",
       "      <th>...</th>\n",
       "      <th>concave_points_mean</th>\n",
       "      <th>concave_points_sd_error</th>\n",
       "      <th>concave_points_worst</th>\n",
       "      <th>symmetry_mean</th>\n",
       "      <th>symmetry_sd_error</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_mean</th>\n",
       "      <th>fractal_dimension_sd_error</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "      <th>diagnosisInt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>247</td>\n",
       "      <td>884626</td>\n",
       "      <td>B</td>\n",
       "      <td>12.89</td>\n",
       "      <td>14.11</td>\n",
       "      <td>84.95</td>\n",
       "      <td>512.2</td>\n",
       "      <td>0.0876</td>\n",
       "      <td>0.1346</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.0398</td>\n",
       "      <td>...</td>\n",
       "      <td>17.7</td>\n",
       "      <td>105.0</td>\n",
       "      <td>639.1</td>\n",
       "      <td>0.1254</td>\n",
       "      <td>0.5849</td>\n",
       "      <td>0.7727</td>\n",
       "      <td>0.1561</td>\n",
       "      <td>0.2639</td>\n",
       "      <td>0.1178</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         ID diagnosis  radius_mean  radius_sd_error  radius_worst  \\\n",
       "247  884626         B        12.89            14.11         84.95   \n",
       "\n",
       "     texture_mean  texture_sd_error  texture_worst  perimeter_mean  \\\n",
       "247         512.2            0.0876         0.1346          0.1374   \n",
       "\n",
       "     perimeter_sd_error  ...  concave_points_mean  concave_points_sd_error  \\\n",
       "247              0.0398  ...                 17.7                    105.0   \n",
       "\n",
       "     concave_points_worst  symmetry_mean  symmetry_sd_error  symmetry_worst  \\\n",
       "247                 639.1         0.1254             0.5849          0.7727   \n",
       "\n",
       "     fractal_dimension_mean  fractal_dimension_sd_error  \\\n",
       "247                  0.1561                      0.2639   \n",
       "\n",
       "     fractal_dimension_worst  diagnosisInt  \n",
       "247                   0.1178             1  \n",
       "\n",
       "[1 rows x 33 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#### Write a function to generate bootstrap samples of the data.\n",
    "\n",
    "import numpy.random as npr\n",
    "n = len(df)\n",
    "samples = []\n",
    "idx = npr.randint(0, n, (100, n))\n",
    "for i, val in enumerate(idx):\n",
    "    samples.append(df.iloc[i,:])\n",
    "df.sample(n=None, frac=None, replace=False, weights=None, random_state=None, axis=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a model to predict the malignant tumors\n",
    "Use at least two classification techniques; compare and contrast the advantages and disadvantages of each."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First Method - Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           B       0.96      0.98      0.97       113\n",
      "           M       0.96      0.91      0.94        58\n",
      "\n",
      "    accuracy                           0.96       171\n",
      "   macro avg       0.96      0.95      0.95       171\n",
      "weighted avg       0.96      0.96      0.96       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df.drop(['diagnosis','diagnosisInt'],axis=1),\n",
    "                                                    df['diagnosis'], test_size=0.30, random_state=301)\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "sc = StandardScaler()\n",
    "sc.fit(X_train)\n",
    "X_train_std = sc.transform(X_train)\n",
    "X_test_std = sc.transform(X_test)\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logmodel = LogisticRegression(C=10.0**-2,solver='liblinear')\n",
    "logmodel.fit(X_train_std,y_train)\n",
    "predictions = logmodel.predict(X_test_std)\n",
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(y_test,predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Avoid overfitting by choosing the regularlization parameter that does best on the test data\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "regularizationParam, auc, log_loss_difference = [], [], []\n",
    "\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "for c in np.arange(-5, 5):\n",
    "    logmodel = LogisticRegression(C=10.0**c, random_state=0,solver='liblinear')\n",
    "    logmodel.fit(X_train_std, y_train)\n",
    "    regularizationParam = np.append(regularizationParam,c)\n",
    "    auc = np.append(auc,roc_auc_score(y_test,logmodel.predict_proba(X_test_std)[:,1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph below shows the highest auc on the test data with a regularization parameter of c = -2. We look at the auc on the test data rather than the training data, because a high AUC on the test data could be due to overfitting. By looking at the test data, we see the AUC on data the model wasn't exposed to when we fit the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1d5418dfcc8>]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3wU9b3/8ddHICIWRQVRwWvVaqooNkFrtRAvPaAeLoqIt3ppxRvHVkUqUupDWo4VoxxbKEotKtWfmIOiaOtBivFSrwkKKAY0VauARayA4qUY+Pz++M42SwywIZvM7s77+Xjsg53Zmcln96HzmfnMzOdr7o6IiCTPNnEHICIi8VACEBFJKCUAEZGEUgIQEUkoJQARkYRqG3cATdG5c2ffZ5994g5DRCSvzJs37yN379Jwfl4lgH322Yfq6uq4wxARyStm9vfG5qsEJCKSUEoAIiIJpQQgIpJQSgAiIgmlBCAiklBKANKqxo+HysqN51VWhvki0rqUAKRVlZbCkCH1SaCyMkyXlsYbl0gSKQFIqyorg4qKsNM/4QT4z/+ECy6A1atDMnj1VXj33TC9YUPLxaEzEZE8exBMCkNZGfTsCXPmhOmbb258OTPYcUfo1Cm8dtpp43+39H677cI2GpM6E6moCPGkzkQqKlrmO4vkIiUAaXVz5sDcubDXXvDZZ/Db30JxMaxaFY78V69u/P2qVfDWW/XvP/ts83+nXbtNJ4mddoJBg6B/fxg8GB57rD4ZiCSFEoC0qspKOO20UN753e+gQ4eNj8Sb4quvYM2ajRPE5hLI6tX15aVVq8L6AHffDWedpZ2/JI8SgLSql1+Grl2he3fo1w+22Sbs/Kuqmr4DbtcOOncOr6Zyh//7PzjnnDA9fTqcfTacdFLTtyWSrzK6CGxmfc1siZnVmtm1jXy+t5nNNbOFZvaUmXVP++wmM3s9ep2RNn9fM3vJzN4yswfMrCg7X0lyWWkp1NbC1VeHnT+EHf/Ika0bx1NPwQ9/CDNmwEMPhTOSwYO/fmFYpJBtMQGYWRtgEtAPKAbONLPiBouVA9PcvQcwFrgxWvdk4AjgcOBI4Boz2yFa5yZggrsfAKwCftT8ryO5rrw8nAGcfXa8cVRV1ZedeveGiy6CL7+EmTPjjUukNWVyBtALqHX3t919HTAdGNBgmWJgbvS+Mu3zYuBpd69z98+ABUBfMzPgOGBGtNw9wMCt/xqSD15/HR5/HK64Atq3jzeWkSM3LjmNHx8S07PP1l8bECl0mSSAbsD7adNLo3npFgCnRe8HAR3NbJdofj8z62BmnYEyYE9gF2C1u9dtZpsAmNkwM6s2s+qVK1dm8p0kR91yS7joe8klcUfydZ06wcSJMH8+TJgQdzQirSOTBNDYndTeYHoE0NvMXgV6A8uAOnd/Avgz8DxwP/ACUJfhNsNM9ynuXuLuJV26fG1AG8kTy5fDfffBj34EO+8cdzSNO/VUGDgQrr8+XKcQKXSZJIClhKP2lO7A8vQF3H25u5/q7j2B0dG8NdG/49z9cHc/kbDjfwv4COhkZm03tU0pLL/9LaxfDz/9adyRbJpZOAsoKoKLLw53CokUskwSQBVwQHTXThEwFJiVvoCZdTaz1LZGAVOj+W2iUhBm1gPoATzh7k64VjA4Wuc84JHmfhnJTZ9+CrffHu7/32+/uKPZvG7d4Kab4Mknw/MBIoVsiwkgqtMPB2YDNUCFuy8ys7Fm1j9arA+wxMzeBLoC46L57YBnzewNYApwTlrd/2fAVWZWS7gm8IcsfSfJMVOnhoevrr467kgyM2wYHHNMiHfFirijEWk55nl0nltSUuIaFD6/1NXB/vuHtg/PPBN3NJlbvBgOOyy0i5g+Pe5oRJrHzOa5e0nD+eoGKi1qxgz4+9/hmmvijqRpDjoIfv5zeOCB0CdIpBDpDEBajDuUlISmbW+8Uf/kb75Ytw6OOCL0G3rjDejYMe6IRLaOzgCk1T39NLzyysZtH/JJURHceScsWwbXXRd3NCLZl4f/W0q+KC+HXXeFc8+NO5Ktd9RRMHw4TJoEL7wQdzQi2aUEIC3ijTfgT38KO8+42z4017hxoXvpRReFspBIoVACkBZx661hRK5LL407kubr2DGMXbBoUXhGQKRQKAFI1n3wAfzxj2Gs363p1Z+LTjkFzjgDfvUrqKmJOxqR7FACkKybODF01Lzqqrgjya7bboPttw+loJYcsF6ktSgBSFatXQuTJ4fGat/8ZtzRZFfXrqG09dxzMGVK3NGINJ8SgGTVXXeF8XZHjIg7kpZx3nlw/PFhPIFly+KORqR5lAAka+rqQi/9730v3D5ZiMzgjjtCievyy9UxVPKbEoBkzcyZ8M47hXv0n/LNb8INN8Ajj4TxhEXylVpBSFa4w5FHhq6fNTXQpk3cEbWsujro1Svc8VRTE0YUE8lVagUhLerZZ8NA61ddVfg7f4C2bUObiJUrw/UAkXykBCBZUV4e7vk/77y4I2k9RxwBV14Jv/996Hskkm+UAKTZFi+GRx8NbR+22y7uaFrXDTeEUc4uugi+/DLuaESaRglAmu3WW0O/n8suizuS1tehQxju8q234Je/jDsakaZRApBmWbECpk2D88+HLl3ijiYeJ54YSl/jx8PChXFHI5I5JQBplokTQ4fMK6+MO5J43XIL7LQT/PjHsH593NGIZEYJQLbaZ5+FLpkDBsCBB8YdTbx22SX0CqqqCklRJB8oAchWu/tu+Pjjwn/wK1NDh8JJJ8Ho0WEcZJFcpwQgW2X9+nDx97vfDa0fJLSJmDw5vL/0UrWJkNynBCBb5eGH4e23dfTf0F57wX//Nzz+ONx/f9zRiGyeWkFIk7mHI/+PPoIlS5Lx5G9TrF8fzor+9rfQJqJQBsWR/KVWEJI1zz0HL72UnLYPTdWmTXg6ePVquPrquKMR2TQlAGmy8vJw18v558cdSe469FC49trwjMQTT8QdjUjjlACkSZYsgVmzwlO/HTrEHU1uGz0avvUtuOSScMusSK7JKAGYWV8zW2JmtWZ2bSOf721mc81soZk9ZWbd0z4bb2aLzKzGzH5jZhbNfyra5vzotWv2vpa0lAkToKgoDIYim9e+fRg68p134Prr445G5Ou2mADMrA0wCegHFANnmllxg8XKgWnu3gMYC9wYrXs08D2gB3AIUAr0TlvvbHc/PHp92NwvIy3rww/hnntC24OuXeOOJj98//swbFhInPPmxR2NyMYyOQPoBdS6+9vuvg6YDgxosEwxMDd6X5n2uQPtgSJgW6AdsKK5QUs8fve70PHyqqvijiS/3HRTSJg//nEYSlIkV2SSALoB76dNL43mpVsAnBa9HwR0NLNd3P0FQkL4IHrNdveatPXuiso/Y1KloYbMbJiZVZtZ9cqVKzMIV1rC55+HFgf9+4e6tmSuUyeYNAnmzw9nAiK5IpME0NiOueHDAyOA3mb2KqHEswyoM7P9gYOB7oSkcZyZfT9a52x3PxQ4Nnqd29gfd/cp7l7i7iVdktpuMgfccw/885968GtrDRoUXtdfD7W1cUcjEmSSAJYCe6ZNdweWpy/g7svd/VR37wmMjuatIZwNvOjua919LfA4cFT0+bLo30+B/0coNUkOSrV96NULjjkm7mjy18SJ4QL6xRerTYTkhkwSQBVwgJnta2ZFwFBgVvoCZtbZzFLbGgVMjd6/RzgzaGtm7QhnBzXRdOdo3XbAKcDrzf860hJmzQpHrSNGhH43snX22COMGfDkk6GRnkjctpgA3L0OGA7MBmqACndfZGZjzax/tFgfYImZvQl0BcZF82cAfwNeI1wnWODujxIuCM82s4XAfELJ6PdZ+1aSVeXlsO++oYQhzXPRRXDsseEJ4RW6HUJipl5AslnPPx/62vz2t2HMX2m+JUugR4+QUKdPjzsaSQL1ApKtUl4eRrq64IK4Iykc3/oWjBkDDzwAjz0WdzSSZEoAsklvvRXaPl92GWy/fdzRFJaRI+GQQ8K4AZ9+Gnc0klRKALJJEyZAu3Yq/bSEoiK4805Ytgyuuy7uaCSplACkUStXwl13wbnnwm67xR1NYTrySDj66HB76Asv1M+vrAx3C4m0NCUAadTkyWr70BpGjYJttoGzzoJ168LOf8gQKC2NOzJJAiUA+ZovvghHpSefDMUN2/5JVp18MvzqV/Duu/CDH4Sdf0UFlJXFHZkkgRKAfM0f/xhKQNdcE3ckyTBqFBx8MDz9dHhOQDt/aS1KALKRDRvgllugpCS0MpaWV1kJy6PmKpMmhWmR1qAEIBt59FF48021fWgtqZr/jBlhmM3S0jCtJCCtQQlANlJeDnvvDaedtuVlpfmqqkLN/4QT4NRT4eWX4d57w3yRlqYEIP/24ovw17/ClVdC27ZxR5MMI0fW1/xPPz08FPbll2G+SEtTApB/u+WWMHjJhRfGHUkylZWFMlBFRdyRSFIoAQgAf/sbPPQQXHIJdOwYdzTJ1LZtaBD36KPhLECkpSkBCBDaPrRpA1dcEXckyTZkSCgDzZ4ddySSBEoAwj//CVOnwjnnwO67xx1NsqXKQP/7v3FHIkmgBCBMnhye/r366rgjkVQZaNYslYGk5SkBJNyXX4bBXvr1g29/O+5oBOrvBlIZSFqaEkDC3XsvfPhhePBLcoPKQNJalAASLNX2oWdP9Z/JJe3aqQwkrUMJIMH+9CdYvDg0fVPbh9yiMpC0BiWABCsvh732gsGD445EGlIZSFqDEkBCvfwyPPMM/PSnoeQguUVlIGkNSgAJdcstsOOO8OMfxx2JbEqqDPTEE3FHIoVKCSBBxo8PbYbfeSe0H774Yqiu1vizuaqsDHbeWb2BpOWo52OCpHrNH3tsGIf2O9+pH4JQck+7dqFF9AMPhDJQ+/ZxRySFRmcACVJWFnr+zJwZxvq9/HKNP5vrVAaSlpRRAjCzvma2xMxqzezaRj7f28zmmtlCM3vKzLqnfTbezBaZWY2Z/cYs3HBoZt8xs9eibf57vrSctWvhppvCkeTChXDppdr55zqVgaQlbTEBmFkbYBLQDygGzjSz4gaLlQPT3L0HMBa4MVr3aOB7QA/gEKAU6B2tMxkYBhwQvfo298vIprnDBRfAokWw7bYwZkzoAaShB3Ob7gaSlpTJGUAvoNbd33b3dcB0YECDZYqBudH7yrTPHWgPFAHbAu2AFWa2O7CDu7/g7g5MAwY265vIZv361+HCb4cOoQQ0dmw4qtT4s7kv1SJaZSDJtkwSQDfg/bTppdG8dAuA1Ciyg4COZraLu79ASAgfRK/Z7l4Trb90C9sEwMyGmVm1mVWvXLkyg3CloT//GUaPhsMOC0eSqbJPWVlIAhp/NrelykB6KEyyLZO7gBqrzXuD6RHARDM7H3gGWAbUmdn+wMFA6prAHDP7PvBFBtsMM92nAFMASkpKGl1GNu2tt+Css8LO/7nnwhlAurIyXQfIdakyUEWF7gaS7MrkDGApsGfadHdgefoC7r7c3U91957A6GjeGsLZwIvuvtbd1wKPA0dF2+y+uW1K8336KQwcGHrMz5z59Z2/5A/dDSQtIZMEUAUcYGb7mlkRMBSYlb6AmXU2s9S2RgFTo/fvAb3NrK2ZtSNcAK5x9w+AT83sqOjunx8Cj2Th+0hkwwb44Q9hyZJwH/k++8QdkTTHccepDCTZt8UE4O51wHBgNlADVLj7IjMba2b9o8X6AEvM7E2gKzAumj8D+BvwGuE6wQJ3fzT67FLgTqA2WubxrHwjAWDcOHj4Ybj5Zjj++LijkebS3UDSEizchJMfSkpKvLq6Ou4wct6jj8KAAXD22TBtmlo9F4rZs6FvX3jkEejff8vLi6SY2Tx3L2k4X08CF5glS8Lg7j17wpQp2vkXEpWBJNuUAArImjXhyH/bbcNF3+22izsiyab0MtC//hV3NFIIlAAKxIYNcO65UFsbbhfca6+4I5KWcPrp8MknuhtIskMJoECMHRtq/xMmQJ8+cUcjLSVVBlJvIMkGJYAC8PDDcMMNcN55MHx43NFIS1IZSLJJCSDPvfFGKP2UlsLtt+uibxKoDCTZogSQx1avDk/6dugADz2kFgFJoTKQZItGBMtT69eH+/zfeQeefBK6d9/yOlIY2rULiX/GjFAG2nbbuCOSfKUzgDx1/fWhy+dtt4UhHiVZhgxRGUiaTwkgDz34YGj18KMfhVG9JHn0UJhkgxJAnnn99XC3z1FHwaRJuuibVKky0COP6G4g2XpKAHlk1arwP33HjuEsQLXfZNPdQNJcSgB5Yv16OPNMeO+9sPPfY4+4I5K4HX887LSTykCy9ZQA8sTPfx66QU6cCEcfHXc0kgtSD4WpDCRbSwkgD1RUhEHdL74Yhg2LOxrJJSoDSXMoAeS4hQvhggvCUf9vfhN3NJJrVAaS5lACyGEffxwu+nbqFB76KSqKOyLJNSoDSXMoAeSoujoYOhSWLQttHnbfPe6IJFelykBz5sQdieQbJYAcdd114X/oyZPhyCPjjkZyWaoMpN5A0lRKADno/vvDYO6XXQYXXhh3NJLr9FCYbC0lgBwzf35o8XDssWFwF5FMpHoDqQwkTaEEkEM++igcye2yS7irQxd9JVMqA8nWUDvoHFFXB2ecAf/4Bzz7LHTtGndEkk9SZaAHH1SLaMmczgByxMiRoa//HXeE0b1EmkplIGkqJYAccO+9od5/xRWh06fI1tBDYdJUSgAxe+UVuOgi6NMHysvjjkbyWaoM9PDDuhtIMpNRAjCzvma2xMxqzezaRj7f28zmmtlCM3vKzLpH88vMbH7a60szGxh9dreZvZP22eHZ/Wq578MPw/+wu+4aLt61axd3RJLv9FCYNMUWE4CZtQEmAf2AYuBMMytusFg5MM3dewBjgRsB3L3S3Q9398OB44DPgfS2VdekPnf3+c3/Ovnjq69CzXblSpg5E7p0iTsiKQQqA0lTZHIG0Auodfe33X0dMB0Y0GCZYmBu9L6ykc8BBgOPu/vnWxtsIRkxAp5+Gu68E444Iu5opFAUFemhMMlcJgmgG/B+2vTSaF66BcBp0ftBQEcz26XBMkOB+xvMGxeVjSaYWWJuXLv77tDZ86qr4Oyz445GCs3pp8OaNSoDyZZlkgAaG3XWG0yPAHqb2atAb2AZUPfvDZjtDhwKzE5bZxRwEFAK7Az8rNE/bjbMzKrNrHrlypUZhJt7xo+HysrwvqoKLrkEevZU2UdaxvHHhw6yKgPJlmSSAJYCe6ZNdweWpy/g7svd/VR37wmMjuatSVtkCDDT3b9KW+cDD/4F3EUoNX2Nu09x9xJ3L+mSp3vM0tJQ73/wwdC6t1OnMLSjmrxJSygqUotoyUwmTwJXAQeY2b6EI/uhwFnpC5hZZ+Bjd99AOLKf2mAbZ0bz09fZ3d0/MDMDBgKvb91X2LTx48POt6ysfl5lZTgKHzmyedvesCHcbbF6dXitWvX19+nzunYNSWCbbeoHdU+PSySbTj8d7roL/vIXOPnkuKORXLXFBODudWY2nFC+aQNMdfdFZjYWqHb3WUAf4EYzc+AZ4PLU+ma2D+EM4ukGm77PzLoQSkzzgUua/W0aSB15V1SEnW1lZf20O3zxxaZ33pvakafer1kTtrEpZrDjjuGOjE6dwq2e69fD4sUwfLh2/tKyUmWgigolANk0883txXJMSUmJV1dXN2mdyko45RTYbbdQdtltN1i3LuzIv/pq8+tuv334n6hTp/odefr7xual3nfsGI720+MYMgQuvTT0+E8lJZGWcsEF4RbjFSvUGyjpzGyeu5c0nF/wzeDKyqC4GKqrYb/9wlnBlnbenTqFo/dsdeNMP/MoKwuv9GmRljBkSLjjTGUg2ZSCTwCVlfDuuzBmTDjyvvji1t/pVlVtvLMvKwvTVVVKANJyVAaSLSnoElDDI++G0yKFTmUggU2XgAq6GdzmjrxFkmDIkHDDwl/+EnckkosK+gxAJOnWrQu3IA8YEK4HSDIl8gxAJOlSvYHUIloaowQgUuBSvYFUBpKGlABECtwJJ6g3kDROCUCkwKWXgdatizsaySVKACIJoBbR0hglAJEEUBlIGqMEIJIAKgNJY5QARBJCZSBpSAlAJCFUBpKGlABEEqKoKDwR/MgjKgNJoAQgkiBDhoQBjfRQmIASgEiipMpAFRVxRyK5QAlAJEFUBpJ0SgAiCXP66SoDSaAEIJIwJ54YhjzV3UCiBCCSMHooTFKUAEQSSGUgASUAkURSGUhACUAkkVQGElACEEkslYFECUAkoVQGEiUAkYRSGUgySgBm1tfMlphZrZld28jne5vZXDNbaGZPmVn3aH6Zmc1Pe31pZgOjz/Y1s5fM7C0ze8DMirL71URkS1QGSrYtJgAzawNMAvoBxcCZZlbcYLFyYJq79wDGAjcCuHulux/u7ocDxwGfA09E69wETHD3A4BVwI+y8H1EpAlUBkq2TM4AegG17v62u68DpgMDGixTDMyN3lc28jnAYOBxd//czIyQEGZEn90DDGxq8CLSPKneQCoDJVMmCaAb8H7a9NJoXroFwGnR+0FARzPbpcEyQ4H7o/e7AKvdvW4z2wTAzIaZWbWZVa9cuTKDcEWkKVItoufO3fKyUlgySQDWyDxvMD0C6G1mrwK9gWVAaueOme0OHArMbsI2w0z3Ke5e4u4lXbp0ySBcEWmKVBlILaKTJ5MEsBTYM226O7A8fQF3X+7up7p7T2B0NG9N2iJDgJnu/lU0/RHQyczabmqbItI6VAZKrkwSQBVwQHTXThGhlDMrfQEz62xmqW2NAqY22MaZ1Jd/cHcnXCsYHM06D3ik6eGLSDak7gZSGShZtpgAojr9cEL5pgaocPdFZjbWzPpHi/UBlpjZm0BXYFxqfTPbh3AG8XSDTf8MuMrMagnXBP7QrG8iIltNdwMlk4WD8fxQUlLi1dXVcYchUnDGjw9H/y+/DCtWhLJQZSVUVcHIkXFHJ81lZvPcvaThfD0JLCKUlsKLL9aXgSorw91BpaVxRyYtqe2WFxGRQldWFu4C6tcPLrkEPvsslIPKyuKOTFqSzgBEBID/+A846SR47z344gv4+OO4I5KWpgQgIkAo+7z0Elx8cbgddPBgOPNM+Oc/445MWooSgIj8u+ZfUQG33w5//jN06BDKQN/+dnhGQAqPEoCIUFUVdv6pmv+JJ8Jjj8F//RfssQcMGgTnnKOyUKHRbaAisllffQU33gi//CV07gx33AH9+295Pckdug1URLZKu3bwi1+Es4Rddw1tI847D1atijsyaS4lABHJyOGHhyTwi1/AfffBIYeEawWSv5QARCRjRUVwww3hbqGdd4aTT4YLLwwPkEn+UQIQkSb7zneguhquuw7uuQcOPRRmz97yepJblABEZKtsuy2MGxdaSOywA/TtCxddBJ98EndkkiklABFpltJSmDcPfvYzmDo1XBuYMyfuqCQTSgAi0mzt28Ovfw3PPx8eIPvBD0JPoU8/jTsy2RwlABHJmiOPhFdfhREjYMqUcG3gySfjjko2RQlARLJqu+3g5pvhr38Ndw0dfzxcfjmsXRt3ZNKQEoCItIijj4b58+HKK2HyZOjRA55uOC6gxEoJQERaTIcOcOutYce/zTbQpw/85CdhvAGJnxKAiLS4Y4+FBQvgiivgN7+Bww6DZ5+NOypRAhCRVrH99nDbbfDUU7BhA/TuDVddBZ9/HndkyaUEICKtqndvWLgQLrsMJkwIPYaefz7uqJJJCUBEWt03vgETJ4YB6Netg2OOgWuuCUNRSutRAhCR2Bx3HLz2GgwbBuXlsNdeMGnSxstUVsL48fHEV+iUAEQkVh07hmEon3gCzGD4cBg6NFwbSA1VWVoad5SFqW3cAYiIQBiGsrY2DET/wANhHGKzcBtpnz5xR1eYdAYgIjljhx3gT3+C88+Hf/0L6urCxeLDDoP/+R9YuTLuCAtLRgnAzPqa2RIzqzWzaxv5fG8zm2tmC83sKTPrnvbZXmb2hJnVmNkbZrZPNP9uM3vHzOZHr8Oz9aVEJH9VVoYB6ceMgR13DE8Sb7dd+LdbNxg8OIxEVlcXd6T5b4slIDNrA0wCTgSWAlVmNsvd30hbrByY5u73mNlxwI3AudFn04Bx7j7HzL4BbEhb7xp3n5GNLyIi+S9V86+ogLKy8EpNd+kCd90Ff/wjPPgg7LFHGJv4wgth//3jjjw/ZXIG0Auodfe33X0dMB0Y0GCZYmBu9L4y9bmZFQNt3X0OgLuvdXc99iEijaqqqt/5Q/i3oiLMP+QQuOUWWLoUHnoIjjgCbroJDjggPFtwzz1qMdFUmSSAbsD7adNLo3npFgCnRe8HAR3NbBfgQGC1mT1kZq+a2c3RGUXKuKhsNMHMtm3sj5vZMDOrNrPqlSoAihS0kSPrd/4pZWVhfkpREQwaBI8+Cu+/DzfeCB98EK4b7LZbGJXshRfAvVVDz0uZJABrZF7Dn3YE0NvMXgV6A8uAOkKJ6djo81JgP+D8aJ1RwEHR/J2BnzX2x919iruXuHtJly5dMghXRJJijz3g2mthyZLQW+j00+H++0Mn0uLi0JZ6xYq4o8xdmSSApcCeadPdgeXpC7j7cnc/1d17AqOjeWuidV+Nykd1wMPAEdHnH3jwL+AuQqlJRKTJzMLTxFOnhrOBP/wBdt45nDl06wYDB8KsWfDVV3FHmlsySQBVwAFmtq+ZFQFDgVnpC5hZZzNLbWsUMDVt3Z3MLHXofhzwRrTO7tG/BgwEXm/OFxERgfBg2YUXwnPPQU0NXH11GLh+wADYc8+QFBYvjjvK3LDFBBAduQ8HZgM1QIW7LzKzsWbWP1qsD7DEzN4EugLjonXXE8o/c83sNUI56ffROvdF814DOgO/ytq3EhEBDjooXCh+//1wBvDd74YGdAcfHMpEf/hDssctNs+jKyUlJSVeXV0ddxgiksdWrIB77w07/5qaMGjNkCHhrOGYY0I5qdCY2Tx3L/nafCUAEUkid3jppXDdYPr0cCaw//4hEaxdCyecsPEdSZWV4XbU9DuS8oUSgIjIJnz2WXi4bOrUMHylGbRrB6NGhU6lixfDGWds/IxCPlECEBHJQG0t3H033HEHfPRRmIamE1kAAARDSURBVGcG3/oWHHVUuH5w8MHh+sJ++0GbNpvdXE5QAhARaYL160M5aNo0KCkJdxfV1MA//lG/TFERHHhgfVJIvQ48MPQvyhWbSgBqBy0i0ohnnglN58aMgcmT68s/q1eHklBNTf3rlVdCCWlD1OnMDPbdN5wlNEwOO+0U7/dKpwQgItLA5prSlZWFUtBRR228zpdfwptvfj05zJ0bWlundO1aX0JKTwzdum18B9L48WEgnJa8EK0EICLSwOaa0m3qInD79tCjR3ilW78e3n03JIP05DB9ejibSOnYsT4pHHRQaHd92mnh755wwsZJKVt0DUBEJAbu4ZmEVEJITw7Llm28bOfOIZE8+ODW3YWkawAiIjnELHQv3W23r+/UP/mkPiFMmQLPPw8jRmT/FlQNCSkikmN22AF69YK99grXFcaMCbemVlZm9+8oAYiI5KD0mv/YseHfIUOymwSUAEREctDmLkRniy4Ci4gUuE1dBNYZgIhIQikBiIgklBKAiEhCKQGIiCSUEoCISELl1V1AZrYS+HvccTRTZ+CjuIPIEfotNqbfY2P6Peo197fY2927NJyZVwmgEJhZdWO3YyWRfouN6ffYmH6Pei31W6gEJCKSUEoAIiIJpQTQ+qbEHUAO0W+xMf0eG9PvUa9FfgtdAxARSSidAYiIJJQSgIhIQikBxMjMRpiZm1nnuGOJi5ndbGaLzWyhmc00s05xxxQHM+trZkvMrNbMro07nriY2Z5mVmlmNWa2yMx+EndMucDM2pjZq2b2WDa3qwQQEzPbEzgReC/uWGI2BzjE3XsAbwKjYo6n1ZlZG2AS0A8oBs40s+J4o4pNHXC1ux8MHAVcnuDfIt1PgJpsb1QJID4TgJFAoq/Cu/sT7l4XTb4IdI8znpj0Amrd/W13XwdMBwbEHFMs3P0Dd38lev8pYafXLd6o4mVm3YGTgTuzvW0lgBiYWX9gmbsviDuWHHMh8HjcQcSgG/B+2vRSEr7TAzCzfYCewEvxRhK7/yEcLG7I9obbZnuDEpjZX4DdGvloNHAd8IPWjSg+m/st3P2RaJnRhNP/+1ozthxhjcxL9JmhmX0DeBD4qbt/Enc8cTGzU4AP3X2emfXJ9vaVAFqIu5/Q2HwzOxTYF1hgZhBKHq+YWS93/0crhthqNvVbpJjZecApwPGezAdTlgJ7pk13B5bHFEvszKwdYed/n7s/FHc8Mfse0N/MTgLaAzuY2b3ufk42Nq4HwWJmZu8CJe6eyK6HZtYXuBXo7e4r444nDmbWlnAB/HhgGVAFnOXui2INLAYWjoruAT5295/GHU8uic4ARrj7Kdnapq4BSNwmAh2BOWY238xujzug1hZdBB8OzCZc9KxI4s4/8j3gXOC46L+H+dHRr7QAnQGIiCSUzgBERBJKCUBEJKGUAEREEkoJQEQkoZQAREQSSglARCShlABERBLq/wNuNCmbQ2+TlAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot(regularizationParam, auc, color='blue', marker='x', label='mylabel')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating Model Fit by Displaying a confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[111,   2],\n",
       "       [  5,  53]], dtype=int64)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logmodel = LogisticRegression(C=10.0**-2, random_state=0,solver='liblinear')\n",
    "logmodel.fit(X_train_std, y_train)\n",
    "predictions = logmodel.predict(X_test_std)\n",
    "confusion_matrix(y_test, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           B       0.96      0.98      0.97       113\n",
      "           M       0.96      0.91      0.94        58\n",
      "\n",
      "    accuracy                           0.96       171\n",
      "   macro avg       0.96      0.95      0.95       171\n",
      "weighted avg       0.96      0.96      0.96       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#### Classification report\n",
    "print(classification_report(y_test,predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In each model, identify the most important predictive variables and explain how you identified them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the absolute value of the coefficients, convavity_worst, concave_points_std_error, and fractal_dimension_mean are the three most important variables. Not surprisingly, the ID label is of low importance - this should just be a random ordering of the observations, so it would be surprising if it were predictive.\n",
    "\n",
    "Full list of coeifficients sorted by importance below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>abs coef</th>\n",
       "      <th>coef</th>\n",
       "      <th>odds ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>concavity_worst</td>\n",
       "      <td>0.230914</td>\n",
       "      <td>0.230914</td>\n",
       "      <td>1.259750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>concave_points_sd_error</td>\n",
       "      <td>0.223046</td>\n",
       "      <td>0.223046</td>\n",
       "      <td>1.249879</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>fractal_dimension_mean</td>\n",
       "      <td>0.218217</td>\n",
       "      <td>0.218217</td>\n",
       "      <td>1.243857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>concave_points_worst</td>\n",
       "      <td>0.210356</td>\n",
       "      <td>0.210356</td>\n",
       "      <td>1.234117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>perimeter_sd_error</td>\n",
       "      <td>0.209471</td>\n",
       "      <td>0.209471</td>\n",
       "      <td>1.233026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>radius_mean</td>\n",
       "      <td>0.205206</td>\n",
       "      <td>0.205206</td>\n",
       "      <td>1.227778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>radius_worst</td>\n",
       "      <td>0.203965</td>\n",
       "      <td>0.203965</td>\n",
       "      <td>1.226255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>texture_mean</td>\n",
       "      <td>0.197671</td>\n",
       "      <td>0.197671</td>\n",
       "      <td>1.218562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>concave_points_mean</td>\n",
       "      <td>0.183726</td>\n",
       "      <td>0.183726</td>\n",
       "      <td>1.201687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>area_sd_error</td>\n",
       "      <td>0.178114</td>\n",
       "      <td>0.178114</td>\n",
       "      <td>1.194961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>perimeter_mean</td>\n",
       "      <td>0.169085</td>\n",
       "      <td>0.169085</td>\n",
       "      <td>1.184221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>fractal_dimension_sd_error</td>\n",
       "      <td>0.164081</td>\n",
       "      <td>0.164081</td>\n",
       "      <td>1.178310</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>symmetry_worst</td>\n",
       "      <td>0.163482</td>\n",
       "      <td>0.163482</td>\n",
       "      <td>1.177604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>smoothness_sd_error</td>\n",
       "      <td>0.159905</td>\n",
       "      <td>0.159905</td>\n",
       "      <td>1.173399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>smoothness_mean</td>\n",
       "      <td>0.154275</td>\n",
       "      <td>0.154275</td>\n",
       "      <td>1.166812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>radius_sd_error</td>\n",
       "      <td>0.152753</td>\n",
       "      <td>0.152753</td>\n",
       "      <td>1.165037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>symmetry_mean</td>\n",
       "      <td>0.146629</td>\n",
       "      <td>0.146629</td>\n",
       "      <td>1.157924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>symmetry_sd_error</td>\n",
       "      <td>0.128255</td>\n",
       "      <td>0.128255</td>\n",
       "      <td>1.136843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>texture_worst</td>\n",
       "      <td>0.105613</td>\n",
       "      <td>0.105613</td>\n",
       "      <td>1.111392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>texture_sd_error</td>\n",
       "      <td>0.094589</td>\n",
       "      <td>0.094589</td>\n",
       "      <td>1.099207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>perimeter_worst</td>\n",
       "      <td>0.085387</td>\n",
       "      <td>0.085387</td>\n",
       "      <td>1.089138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>concavity_sd_error</td>\n",
       "      <td>0.081690</td>\n",
       "      <td>-0.081690</td>\n",
       "      <td>0.921558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>fractal_dimension_worst</td>\n",
       "      <td>0.065184</td>\n",
       "      <td>0.065184</td>\n",
       "      <td>1.067356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>compactness_worst</td>\n",
       "      <td>0.050926</td>\n",
       "      <td>0.050926</td>\n",
       "      <td>1.052245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>area_mean</td>\n",
       "      <td>0.048701</td>\n",
       "      <td>-0.048701</td>\n",
       "      <td>0.952466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>smoothness_worst</td>\n",
       "      <td>0.043384</td>\n",
       "      <td>-0.043384</td>\n",
       "      <td>0.957544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>compactness_mean</td>\n",
       "      <td>0.037250</td>\n",
       "      <td>-0.037250</td>\n",
       "      <td>0.963435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>concavity_mean</td>\n",
       "      <td>0.033120</td>\n",
       "      <td>-0.033120</td>\n",
       "      <td>0.967422</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>compactness_sd_error</td>\n",
       "      <td>0.010113</td>\n",
       "      <td>-0.010113</td>\n",
       "      <td>0.989938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>ID</td>\n",
       "      <td>0.008594</td>\n",
       "      <td>-0.008594</td>\n",
       "      <td>0.991443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>area_worst</td>\n",
       "      <td>0.001431</td>\n",
       "      <td>-0.001431</td>\n",
       "      <td>0.998570</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         label  abs coef      coef  odds ratio\n",
       "21             concavity_worst  0.230914  0.230914    1.259750\n",
       "23     concave_points_sd_error  0.223046  0.223046    1.249879\n",
       "28      fractal_dimension_mean  0.218217  0.218217    1.243857\n",
       "24        concave_points_worst  0.210356  0.210356    1.234117\n",
       "8           perimeter_sd_error  0.209471  0.209471    1.233026\n",
       "1                  radius_mean  0.205206  0.205206    1.227778\n",
       "3                 radius_worst  0.203965  0.203965    1.226255\n",
       "4                 texture_mean  0.197671  0.197671    1.218562\n",
       "22         concave_points_mean  0.183726  0.183726    1.201687\n",
       "11               area_sd_error  0.178114  0.178114    1.194961\n",
       "7               perimeter_mean  0.169085  0.169085    1.184221\n",
       "29  fractal_dimension_sd_error  0.164081  0.164081    1.178310\n",
       "27              symmetry_worst  0.163482  0.163482    1.177604\n",
       "14         smoothness_sd_error  0.159905  0.159905    1.173399\n",
       "13             smoothness_mean  0.154275  0.154275    1.166812\n",
       "2              radius_sd_error  0.152753  0.152753    1.165037\n",
       "25               symmetry_mean  0.146629  0.146629    1.157924\n",
       "26           symmetry_sd_error  0.128255  0.128255    1.136843\n",
       "6                texture_worst  0.105613  0.105613    1.111392\n",
       "5             texture_sd_error  0.094589  0.094589    1.099207\n",
       "9              perimeter_worst  0.085387  0.085387    1.089138\n",
       "20          concavity_sd_error  0.081690 -0.081690    0.921558\n",
       "30     fractal_dimension_worst  0.065184  0.065184    1.067356\n",
       "18           compactness_worst  0.050926  0.050926    1.052245\n",
       "10                   area_mean  0.048701 -0.048701    0.952466\n",
       "15            smoothness_worst  0.043384 -0.043384    0.957544\n",
       "16            compactness_mean  0.037250 -0.037250    0.963435\n",
       "19              concavity_mean  0.033120 -0.033120    0.967422\n",
       "17        compactness_sd_error  0.010113 -0.010113    0.989938\n",
       "0                           ID  0.008594 -0.008594    0.991443\n",
       "12                  area_worst  0.001431 -0.001431    0.998570"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coef_df = pd.DataFrame({'label': X_test.columns, 'abs coef': abs(logmodel.coef_[0]), 'coef': logmodel.coef_[0], 'odds ratio': np.exp(logmodel.coef_[0])}, columns=['label', 'abs coef', 'coef', 'odds ratio'])\n",
    "coef_df.sort_values(by=['abs coef'], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### second method - random forest\n",
    "We use gridsearch to choose the hyperparameters that give the best result on crossvalidation. This allows us to choose hyperparameters that perform best on the test data, hence minimizing overfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'criterion': 'entropy',\n",
       " 'max_depth': 6,\n",
       " 'max_features': 'auto',\n",
       " 'n_estimators': 200}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rfc=RandomForestClassifier(random_state=42)\n",
    "param_grid = {\n",
    "    'n_estimators': [200, 500],\n",
    "    'max_features': ['auto', 'sqrt', 'log2'],\n",
    "    'max_depth' : [4,5,6,7,8],\n",
    "    'criterion' :['gini', 'entropy']\n",
    "}\n",
    "CV_rfc = GridSearchCV(estimator=rfc, param_grid=param_grid, cv= 5)\n",
    "CV_rfc.fit(X_train_std, y_train)\n",
    "CV_rfc.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the selected hyperparameters to train the final model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='entropy',\n",
       "                       max_depth=6, max_features='auto', max_leaf_nodes=None,\n",
       "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                       min_samples_leaf=1, min_samples_split=2,\n",
       "                       min_weight_fraction_leaf=0.0, n_estimators=200,\n",
       "                       n_jobs=None, oob_score=False, random_state=42, verbose=0,\n",
       "                       warm_start=False)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.datasets import make_classification\n",
    "\n",
    "clf =RandomForestClassifier(random_state=42, max_features='auto', n_estimators= 200, max_depth=6, criterion='entropy')\n",
    "clf.fit(X_train_std, y_train)\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.datasets import make_classification\n",
    "clf =RandomForestClassifier(random_state=42, max_features='auto', n_estimators= 200, max_depth=6, criterion='entropy')\n",
    "clf.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Random forest does not perform as well as the logistic model on the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[109,   4],\n",
       "       [  5,  53]], dtype=int64)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = clf.predict(X_test_std)\n",
    "confusion_matrix(y_test, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Identify the most important predictive variables and explain how you identified them.\n",
    "See below, where features are sorted by importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>importances</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>concave_points_sd_error</td>\n",
       "      <td>0.124953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>concave_points_worst</td>\n",
       "      <td>0.118352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>concavity_worst</td>\n",
       "      <td>0.111328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>perimeter_sd_error</td>\n",
       "      <td>0.109609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>fractal_dimension_mean</td>\n",
       "      <td>0.096208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>radius_worst</td>\n",
       "      <td>0.051032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>texture_mean</td>\n",
       "      <td>0.046547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>perimeter_mean</td>\n",
       "      <td>0.043506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>symmetry_worst</td>\n",
       "      <td>0.037537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>smoothness_sd_error</td>\n",
       "      <td>0.036178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>radius_mean</td>\n",
       "      <td>0.027487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>symmetry_sd_error</td>\n",
       "      <td>0.020855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>fractal_dimension_sd_error</td>\n",
       "      <td>0.018051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>area_sd_error</td>\n",
       "      <td>0.017707</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>texture_worst</td>\n",
       "      <td>0.017046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>concave_points_mean</td>\n",
       "      <td>0.014668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>symmetry_mean</td>\n",
       "      <td>0.014251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>smoothness_mean</td>\n",
       "      <td>0.013496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>radius_sd_error</td>\n",
       "      <td>0.011374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>texture_sd_error</td>\n",
       "      <td>0.008563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>concavity_sd_error</td>\n",
       "      <td>0.007161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>fractal_dimension_worst</td>\n",
       "      <td>0.007038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>compactness_sd_error</td>\n",
       "      <td>0.006497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>perimeter_worst</td>\n",
       "      <td>0.006466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>ID</td>\n",
       "      <td>0.006064</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>compactness_mean</td>\n",
       "      <td>0.005919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>compactness_worst</td>\n",
       "      <td>0.005702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>concavity_mean</td>\n",
       "      <td>0.005161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>area_mean</td>\n",
       "      <td>0.004711</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>area_worst</td>\n",
       "      <td>0.003912</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>smoothness_worst</td>\n",
       "      <td>0.002619</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         label  importances\n",
       "23     concave_points_sd_error     0.124953\n",
       "24        concave_points_worst     0.118352\n",
       "21             concavity_worst     0.111328\n",
       "8           perimeter_sd_error     0.109609\n",
       "28      fractal_dimension_mean     0.096208\n",
       "3                 radius_worst     0.051032\n",
       "4                 texture_mean     0.046547\n",
       "7               perimeter_mean     0.043506\n",
       "27              symmetry_worst     0.037537\n",
       "14         smoothness_sd_error     0.036178\n",
       "1                  radius_mean     0.027487\n",
       "26           symmetry_sd_error     0.020855\n",
       "29  fractal_dimension_sd_error     0.018051\n",
       "11               area_sd_error     0.017707\n",
       "6                texture_worst     0.017046\n",
       "22         concave_points_mean     0.014668\n",
       "25               symmetry_mean     0.014251\n",
       "13             smoothness_mean     0.013496\n",
       "2              radius_sd_error     0.011374\n",
       "5             texture_sd_error     0.008563\n",
       "20          concavity_sd_error     0.007161\n",
       "30     fractal_dimension_worst     0.007038\n",
       "17        compactness_sd_error     0.006497\n",
       "9              perimeter_worst     0.006466\n",
       "0                           ID     0.006064\n",
       "16            compactness_mean     0.005919\n",
       "18           compactness_worst     0.005702\n",
       "19              concavity_mean     0.005161\n",
       "10                   area_mean     0.004711\n",
       "12                  area_worst     0.003912\n",
       "15            smoothness_worst     0.002619"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "randomforest_coef_df = pd.DataFrame({'label': X_test.columns, 'importances': clf.feature_importances_}, columns=['label', 'importances'])\n",
    "randomforest_coef_df.sort_values(by=['importances'], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## To Technical Audiences\n",
    "#### Explain the limitations of your analysis and identify possible further steps you could take.\n",
    " \n",
    "No instruction was given regarding tradeoff between sensitivity and specificity. In a real world example, we might decide that we'd like to identify a high % of malignancies, even at the cost of incorrectly identifying some benign cases as malignant. We would set a risk threshold based on this decision.\n",
    "\n",
    "More different model types could be tried. We could also try combining measurements (ratios, squares, products) to make other variables and creating binary variables for measurments that could be categorized as outliers.\n",
    "\n",
    "If we look at the boxplot for smoothness_mean, essentially 100% of the high value outliers are for malignancies. We could create an smoothness_mean_high_outlier variable to improve the model.\n",
    "\n",
    "The ID column should be removed from the model, since it is just a random number to identify the row.\n",
    "\n",
    "Before applying this model to other data sets, we'd want to confirm that the sample used to train the model was representive. For example, if this data was created using younger Caucasian women, we might not want to use it to make predictions on image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## To Non-Technical Audiences\n",
    "#### Write a short summary of your analysis, explaining how your model works and how it performs.\n",
    "#### Briefly explain the factors that contributed to malignant vs benign tumor identification.\n",
    "\n",
    "**Executive summary**: When the best model predicts the tumor is malignant, it is correct 96.4% of the time!  And when it predicts the tumor is benign, it is correct 95.7% of the time.\n",
    "\n",
    "For most of the risk factors, the likelihood of malignancy increases. Fractal_dimension_mean, concave_points_sd_error and perimeter_sd_error show the strongest association with malignancy.\n",
    "\n",
    "By looking at muliple measurements for the tumor, the models can better distinguish malignant from benign tumors.\n",
    "\n",
    "For the random forest model, concave_points_sd_error, concave_points_worst, and concavity_worst were the three most imporant predictors. For the logistic model, concavity_worst, concave_points_sd_error and fractal_dimension_mean were the top predictors. (If more time, would do a bar chart here showing variable importance for the top 5 predictors).\n",
    "\n",
    "The logistic model increases the risk for each risk factor, without considering possible interactions between the risk factors, while the random forest model does capture those intractions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}