notebooks/06_01_Machine_learning_supervise.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Chapitre 6 - Différentes utilisations dumachine learning avec Python (1ère partie)\n",
    "\n",
    "## 6. 4 L’apprentissage supervisé avec Scikit-Learn"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Chapitre 6 - Différentes utilisations dumachine learning avec Python (1ère partie)\n",
    "\n",
    "## 6. 4 L’apprentissage supervisé avec Scikit-Learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les méthodes d’apprentissage supervisé sont les méthodes actuellement les plus\n",
    "utilisées en data science. Il s’agit d’essayer de prédire une variable cible et d’utiliser\n",
    "différentes méthodes pour arriver à cette fin.\n",
    "Nous allons illustrer ces méthodes de traitement de données avec du code et des\n",
    "cas pratiques.\n",
    "### 6.4.1 Les données et leur transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ce jeu de données est décrit en détail au début du chapitre 4. Pour rappel, il est composé de 3333 individus et de 18 variables. Il est stocké dans un fichier csv, nommé telecom.csv, accessible dans le répertoire Data. On le récupère en utilisant Pandas :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "churn=pd.read_csv(\"../data/telecom.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ce jeu de données n’a pas de données manquantes et nous allons devoir effectuer\n",
    "quelques transformations pour l’adapter à nos traitements. Nous voyons par exemple qu’il est composé de trois colonnes object.\n",
    "\n",
    "Nous pouvons afficher les statistiques descriptives pour les colonnes object :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>unique</th>\n",
       "      <th>top</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>State</th>\n",
       "      <td>3333</td>\n",
       "      <td>51</td>\n",
       "      <td>WV</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Phone</th>\n",
       "      <td>3333</td>\n",
       "      <td>3333</td>\n",
       "      <td>394-5489</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Int'l Plan</th>\n",
       "      <td>3333</td>\n",
       "      <td>2</td>\n",
       "      <td>no</td>\n",
       "      <td>3010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VMail Plan</th>\n",
       "      <td>3333</td>\n",
       "      <td>2</td>\n",
       "      <td>no</td>\n",
       "      <td>2411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Churn?</th>\n",
       "      <td>3333</td>\n",
       "      <td>2</td>\n",
       "      <td>False.</td>\n",
       "      <td>2850</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           count unique       top  freq\n",
       "State       3333     51        WV   106\n",
       "Phone       3333   3333  394-5489     1\n",
       "Int'l Plan  3333      2        no  3010\n",
       "VMail Plan  3333      2        no  2411\n",
       "Churn?      3333      2    False.  2850"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "churn.describe(include=\"object\").transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On voit que les données sont toutes binaires. \n",
    "\n",
    "Pour les variables binaires, il nous suffit de les recoder avec Scikit-Learn pour obtenir des données exploitables. Par\n",
    "ailleurs, il existe une autre variable qualitative dans notre jeu de données, Area Code,\n",
    "qui est numérique mais avec trois modalités :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "415    1655\n",
       "510     840\n",
       "408     838\n",
       "Name: Area Code, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "churn[\"Area Code\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### La préparation des données\n",
    "\n",
    "Nous allons utiliser le processus de traitement classique pour transformer nos\n",
    "données avec Scikit-Learn. Dans ce cas, nous n’avons pas de données manquantes,\n",
    "nous travaillons donc sur la transformation des variables qualitatives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder, OneHotEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LabelEncoder va nous permettre de transformer les valeurs textuelles en entiers.\n",
    "Nous pouvons utiliser pour chaque variable qualitative :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder=LabelEncoder()\n",
    "churn[\"Churn?\"]=encoder.fit_transform(churn[\"Churn?\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_label_encode={}\n",
    "for col in churn.columns:\n",
    "    if churn[col].dtype == object:\n",
    "        dict_label_encode[col]=LabelEncoder()\n",
    "        churn[col]=dict_label_encode[col].fit_transform(churn[col])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous utilisons la classe OneHotEncoder afin de transformer la colonne Area Code\n",
    "en trois colonnes binaires (le chapitre 4 détaille le fonctionnement de cet outil) :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# on définit l’objet\n",
    "encoder_area=OneHotEncoder(sparse=False,categories=\"auto\")\n",
    "\n",
    "# on crée un array dans lequel on stocke le résultat\n",
    "area=encoder_area.fit_transform(np.array(churn[\"Area Code\"]).reshape(-1, 1))\n",
    "\n",
    "# on définit un DataFrame nettoyé qui consiste en la concaténation du\n",
    "# DataFrame initial et des nouvelles colonnes\n",
    "churn_clean=pd.concat([churn, pd.DataFrame(area,columns=encoder_area.categories_,\n",
    "                                           index=churn.index)], axis=1)\n",
    "# on supprime la colonne initiale\n",
    "churn_clean.drop(\"Area Code\", axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prédire l’attrition des clients\n",
    "Lorsqu’on veut prédire une variable binaire, on devra avoir une colonne du type\n",
    "binaire. On préfère généralement un codage 0/1 afin de garder un type entier simple à gérer. \n",
    "\n",
    "Les variables explicatives x auront été préparées de manière intelligente afin de bien appliquer nos modèles.\n",
    "\n",
    "On crée donc x et y :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=churn_clean.drop(\"Churn?\", axis=1)\n",
    "y=churn_clean[\"Churn?\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.name=\"Churn!\"\n",
    "x.to_csv(\"../data/x_telecom.csv\")\n",
    "y.to_csv(\"../data/y_telecom.csv\",header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pour la séparation, on utilise la fonction train_test_split() de Scikit-Learn.\n",
    "\n",
    "Cette fonction permet de créer automatiquement autant de structures que nécessaire\n",
    "à partir de nos données. \n",
    "\n",
    "Elle utilise une randomisation des individus et ensuite une séparation en fonction d’un paramètre du type test_size :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# on importe la fonction\n",
    "from sklearn.model_selection import train_test_split\n",
    "# dans ce cas on a :\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dans certains cas, il peut arriver qu’il y ait une forte disparité de distribution des\n",
    "modalités entre les proportions d’acceptation et de refus. On peut vouloir faire en\n",
    "sorte que les répartitions des modalités de y soient égales dans les différents échantillons,\n",
    "on pourra alors utiliser une stratification. On va utiliser une stratification en\n",
    "prenant y comme base pour effectuer la stratification :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.855086\n",
       "1    0.144914\n",
       "Name: Churn!, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3,stratify = y)\n",
    "y.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ainsi les deux échantillons _train et _test ont la même distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.855\n",
       "1    0.145\n",
       "Name: Churn!, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.855122\n",
       "1    0.144878\n",
       "Name: Churn!, dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Les données déséquilibrées\n",
    "Ce problème se pose souvent lorsque vous cherchez à identifier des évènements\n",
    "rares, notamment la recherche de fraudes. Dans ce cas, la variable cible y a souvent\n",
    "moins de 1 % d’une des deux modalités."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imblearn.under_sampling import NearMiss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ss_echant=NearMiss()\n",
    "# on applique la méthode fit_sample() sur nos données\n",
    "x_sous, y_sous= ss_echant.fit_sample(x_train, y_train)\n",
    "# on le transforme en Series pour avoir un affichage psl uagréable\n",
    "pd.Series(y_sous).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On voit donc que le nombre de 1 (classe minoritaire) n’a pas changé.\n",
    "Cette étape permet d’obtenir des données x construites de manière à bien différencier\n",
    "les individus.\n",
    "Pour le sur-échantillonnage, l’algorithme SMOTE (Synthetic Minority Oversampling\n",
    "Technique) fait référence. Cet algorithme va permettre de créer des individus proches\n",
    "de ceux de la classe minoritaire afin de sur-représenter cette classe :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imblearn.over_sampling import SMOTE\n",
    "sur_echant=SMOTE()\n",
    "# on applique la méthode fit_sample() sur nos données\n",
    "x_sur, y_sur= sur_echant.fit_sample(x_train, y_train)\n",
    "# on le transforme en Series pour avoir un affichage psl uagréable\n",
    "pd.Series(y_sur).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On voit ici que le nombre d’individus de la classe majoritaire reste le même.\n",
    "Ce package propose aussi une approche intermédiaire qui va combiner du souséchantillonnage\n",
    "et du sur-échantillonnage, et qui s’adapte bien, soit lorsque le jeu\n",
    "de données global est très grand, soit quand la classe sous-représentée a très peu\n",
    "d’occurrences. On utilise une combinaison de SMOTE et de plus proches voisins :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imblearn.combine import SMOTEENN\n",
    "comb_echant=SMOTEENN()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# on applique la méthode fit_sample() sur nos données\n",
    "x_comb, y_comb= comb_echant.fit_sample(x_train, y_train)\n",
    "# on le transforme en Series pour avoir un affichage plus agréable\n",
    "pd.Series(y_comb).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On voit qu’on arrive à un jeu de données intermédiaire.\n",
    "L’approche à privilégier dépendra de vos données. Dans le cadre de notre exemple,\n",
    "nous allons garder la répartition actuelle afin d’illustrer la suite.\n",
    "Nous avons donc quatre jeux de données prêts à être utilisés pour nos traitements.\n",
    "L’étape suivante est le choix de l’algorithme de machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.2 Le choix et l’ajustement de l’algorithme"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ensuite, on crée un objet à partir de la classe du modèle en lui fournissant les\n",
    "hyperparamètres dont il a besoin :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "modele_rf=RandomForestClassifier()\n",
    "modele_knn=KNeighborsClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dans ce cas, on prend les hyperparamètres par défaut.\n",
    "\n",
    "On peut ensuite ajuster notre modèle en utilisant les données :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\s4d-asus-14\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:248: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=None, n_neighbors=5, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "modele_rf.fit(x_train,y_train)\n",
    "modele_knn.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Une fois qu’on a estimé les paramètres du modèle, on va pouvoir extraire des\n",
    "informations. De nouveaux attributs de chaque classe apparaissent, ils se terminent par le symbole underscore _ :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.02794616, 0.02449936, 0.03179185, 0.08035101, 0.02482033,\n",
       "       0.03582286, 0.14214204, 0.03446039, 0.13223307, 0.06482964,\n",
       "       0.02277616, 0.06017793, 0.03089129, 0.02449727, 0.02587158,\n",
       "       0.03248892, 0.03773679, 0.04904695, 0.10345801, 0.00508424,\n",
       "       0.0052742 , 0.00379995])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "modele_rf.feature_importances_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ce qui va nous intéresse avant tout, c’est de prédire avec notre modèle. Pour cela nous allons utiliser la méthode .predict() :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predict_rf = modele_rf.predict(x_test)\n",
    "y_predict_knn = modele_knn.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On obtient ainsi une valeur prédite pour les éléments de notre échantillon de\n",
    "validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cas d’un modèle de régression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sur les données des territoires d’Ile-de-France, nous désirons prédire le salaire\n",
    "médian des communes en fonction des autres variables des données. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=pd.read_csv(\"../data/base-comparateur-de-territoires.csv\",sep=\";\").dropna()\n",
    "# on prépare les données\n",
    "y_quanti=data[\"MED14\"]\n",
    "x_quanti=data.select_dtypes(np.number).drop([\"CODGEO\",\"REG\",\"DEP\",\"MED14\"],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ridge(alpha=1.0, copy_X=True, fit_intercept=True, max_iter=None,\n",
       "   normalize=False, random_state=None, solver='auto', tol=0.001)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# on sépare les échantillons apprentissage\n",
    "x_train_cont, x_test_cont, y_train_cont, y_test_cont = train_test_split(x_quanti,\n",
    "                                                                        y_quanti,\n",
    "                                                                        test_size=0.3)\n",
    "\n",
    "# on utilise un modèle de régression de Ridge\n",
    "modele_ridge = Ridge()\n",
    "\n",
    "# on ajuste le modèle aux données\n",
    "modele_ridge.fit(x_train_cont, y_train_cont)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.04153311e+00, -1.06355478e-01, -5.15482662e+00,  4.91484520e-01,\n",
       "        4.07570804e+00,  3.15050801e-02, -5.14320869e-01, -1.79910671e+01,\n",
       "        1.61751420e-01,  3.15084200e-02,  2.25690764e-01, -9.52801791e-02,\n",
       "        4.82129105e-02, -5.65330772e-01,  4.88506858e+02,  1.67687264e+02,\n",
       "        9.78434416e-01, -8.99834330e-01, -4.19852400e-02, -1.07635603e+00,\n",
       "       -1.91724132e-01, -4.38384184e-01,  1.75462889e+00,  1.10932036e+00,\n",
       "        2.57552469e+00, -2.53682498e+00, -6.07033181e-01,  6.05419591e-02,\n",
       "        1.21364470e+00, -3.54779022e+00, -4.69990926e+00])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# on peut afficher les coefficients du modèle :\n",
    "modele_ridge.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# on prédit grâce à ce modèle\n",
    "y_predict_cont=modele_ridge.predict(x_test_cont)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.3 Les indicateurs pour valider un modèle\n",
    "La partie validation d’un modèle d’apprentissage supervisé est extrêmement\n",
    "importante. L’objectif d’un modèle d’apprentissage supervisé est de prédire une\n",
    "valeur la plus proche possible de la réalité. Nous différencions trois types d’indices\n",
    "en fonction du type de variable cible. Tous les indicateurs de qualité du modèle sont\n",
    "stockés dans le module *metrics* de Scikit-Learn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Le pourcentage de bien classés\n",
    "Il s’agit de l’indicateur le plus connu. On le nomme accuracy. Il est calculé à partir du rapport entre le nombre d’individus bien classés et le nombre total d’individus dans l’échantillon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pourcentage de bien classés pour le modèle RF : 0.94\n",
      "Pourcentage de bien classés pour le modèle kNN : 0.865\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_modele_rf=accuracy_score(y_test,y_predict_rf)\n",
    "accuracy_modele_knn=accuracy_score(y_test,y_predict_knn)\n",
    "print(\"Pourcentage de bien classés pour le modèle RF :\",accuracy_modele_rf)\n",
    "print(\"Pourcentage de bien classés pour le modèle kNN :\",accuracy_modele_knn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### La matrice de confusion\n",
    "Il s’agit d’un autre indicateur important pour juger de la qualité d’un modèle, il n’est pas défini par une seule valeur mais par une matrice dans laquelle on peut lire le croisement entre les valeurs observées et les valeurs prédites à partir du modèle. \n",
    "\n",
    "Pour calculer cette matrice, on pourra utiliser :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion pour le modèle RF :\n",
      "[[846   9]\n",
      " [ 51  94]]\n",
      "Matrice de confusion pour le modèle kNN :\n",
      "[[839  16]\n",
      " [119  26]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix_rf=confusion_matrix(y_test,y_predict_rf)\n",
    "confusion_matrix_knn=confusion_matrix(y_test,y_predict_knn)\n",
    "print(\"Matrice de confusion pour le modèle RF :\",\n",
    "confusion_matrix_rf, sep=\"\\n\")\n",
    "print(\"Matrice de confusion pour le modèle kNN :\",\n",
    "confusion_matrix_knn, sep=\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Le rappel (recall), la précision et le f1-score\n",
    "\n",
    "Scikit-Learn possède des fonctions pour chacun de ces indicateurs, mais il peut\n",
    "être intéressant d’utiliser une autre fonction qui les affiche pour chaque classe :\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rapport pour le modèle RF :\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.97       855\n",
      "           1       0.91      0.65      0.76       145\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      1000\n",
      "   macro avg       0.93      0.82      0.86      1000\n",
      "weighted avg       0.94      0.94      0.94      1000\n",
      "\n",
      "Rapport pour le modèle kNN :\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.97       855\n",
      "           1       0.91      0.65      0.76       145\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      1000\n",
      "   macro avg       0.93      0.82      0.86      1000\n",
      "weighted avg       0.94      0.94      0.94      1000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print(\"Rapport pour le modèle RF :\",\n",
    "      classification_report(y_test,y_predict_rf) ,sep=\"\\n\")\n",
    "print(\"Rapport pour le modèle kNN :\",\n",
    "      classification_report(y_test,y_predict_rf) ,sep=\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### La courbe ROC\n",
    "Nous avons vu que lorsque les classes sont fortement déséquilibrées, la matrice\n",
    "de confusion est parfois dure à interpréter. La courbe ROC (Receiver Operating\n",
    "Characteristic) est là pour combler ce défaut. Elle est en fait la proportion de vrais positifs en fonction de la proportion de faux positifs.\n",
    "\n",
    "Il s’agit en fait de représenter le rappel (recall) en fonction de (1- spécificité) sur une courbe en faisant varier le seuil de classification (c’est-à-dire le point à partir duquel une observation est considérée comme positive).\n",
    "\n",
    "La courbe ROC s’obtient avec Scikit-Learn et Matplotlib :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x2473bb3a2e8>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VMXXwPHvpBcgQEKTAKG3AAlFQFSaICAKKoooCjb8gYjYsYICYnstCCoo2BURFRuIgvQmQYr0XgJiQoBASM/O+8dk0wjJJiS5u9nzeZ59djf37mZySQ6zc2bOKK01Qgghyj8PqxsghBCibEjAF0IINyEBXwgh3IQEfCGEcBMS8IUQwk1IwBdCCDchAV8IIdyEBHwhhHATEvCFEMJNeFn1jUNCQnRYWJhV314IIVzSxo0bT2qtqxXntZYF/LCwMKKioqz69kII4ZKUUoeL+1oZ0hFCCDchAV8IIdyEBHwhhHATEvCFEMJNSMAXQgg3UWjAV0rNVkrFKKW2XeS4UkpNVUrtU0ptVUq1LflmCiGEuFSO9PA/AfoUcLwv0DjzNgJ4/9KbJYQQoqQVGvC11iuAUwWcMgD4TBvrgMpKqVol1UAhhBCGLf7cJb2+JBZe1QaO5ngenfm1fx158cyZ8NVXF3797bchIgIWL4ZJky48PmMGNG0KP/8M//d/Fx7//HOoUwe++Qbez+czx7x5EBICn3xibnktWAABAfDeezB37oXHly0z92+8Ab/8kvuYvz8sXGgeT5wIS5bkPh4cDN99Zx4//TSsXZv7eGgofPGFeTx2LGzenPt4kybmugGMGAF79uQ+HhFhrh/A0KEQHZ37eOfOMGWKeXzzzRAXl/t4z57w/PPmcd++kJSU+3j//vD44+Zxt25c4NZbYdQoSEyEfv0uPD58uLmdPAmDBl14fORIGDwYjh6FO++88Phjj8H118Pu3fDAAxcef+45uOYac93Gjr3w+MsvwxVXwJo18MwzFx6X3z3zWH73LjxeGr976emQlgapqeaaBQaaf5fFi7O/3qSpjYDzcbwTffOFb1oEJRHwVT5fy3dndKXUCMywD3Xr1gUgPBwqVYKzZ0ugJUIIYbGUFDhxAs6dyw7YVaqAry9s327+I77nHnPusWOwb1/2a3P+B+vlBd7e4Od/htR/DhCQdBZVvzLEF79tSut8Y3Puk5QKA37RWofnc2wGsExr/XXm891AN611gT389u3baymtIIRwdjab6YX7+JiO6W+/QYcOUL8+/PMPjB8PMTHZt/h8AvI335hPH2vXwpgxMGsWtG4NW7aYT2HVq2ffvCuk8MueQ9zRoRahX8wi47nnsdlsZLwwHr/HH0X5+GzUWrcvzs9SEj38n4DRSqk5QEcgvrBgn9OaNeb+iitKoCVCCOEAmw08PMyQ0eLFuQN23ltsLEyeDE89BadOmSGfWbNMwE9PN8Na1atDu3a5A3fOW+3a5vt27gwbNmS3o00bc8uwaRKS0wkK8ObYGRvrXv6NEY9+BLu34dmvH57Tp+NdAsUmCw34SqmvgW5AiFIqGhgPeANorT8AFgD9gH1AInB3URpgH8eyj0sKIURRpaebcfn//ss/aLdrZ8b2bTaoXBkeeQRefNEMu9xwQ/b7VKyYHaTr14eOHc1je4c0NBS2bYN69czzyEjz/FLYbJq+76wgvHYQb/ZpQO3xz/P9p9NQNWrAt9+aZIfKb+S86AoN+FrrIYUc18CDJdIaIYQAtDbDJ/aADdCli7mfMMEkvUePNs/r1YMjR/J/Hy8vE7CDgsxzDw94+OHsAB4SYnrc1atDtWom6V0QLy9o2fKSfjQA/jp4ilV7Y3m0d1M8PBR3dqxL6w1Lofl18O+/qFGjzMcKe8NLiGXlkYUQ7mnPHjM00qmTef7mm2YsO2+vPDU1+zWRkfD33+bx6tWQOecDMLNpvLygRo0Lh1IqV76wczxxYvZjDw9oX6zR8KKLS0ihSoAPHh6KqMOn+HL9Ee69sgFBJ//lzlceNtO+WreG7783Hy1KgUNJ29JgT9rap1bJkI4QrunIETh4MDtQ5zeskpoKhw6Z8wcPNrNRdu82z/v1M7NX7EE6v8Bdu3bJ9KytsuHQKW7/cB2zh3fgqsbVSExNx9OWge970+GFF8xHmhdfNB8/vL0LfC+llKVJWyFEOXH+/MWTly+9ZMa433rL3A4fNr3nCRPg44+z38PDwwyV2IO1PZmptTn/uecgOTn7/AULyvzHLHXpGTY+XXuY2pX96RNek9ahQdxzZX3qVQ0EIGDLJrOQYfNmuO46mD49OzFQiiwP+PZFGkKIkmdPZsbEQFiYWfOydSvMmWMWMFWtamacTJpkzklMzP99KlSAhx4yAb9hQ+jTx8wx9/ExndKhQ7N75lWrgqfnxdvUqlWp/KiW01oTcy6FGpX88PRQfLPhCG3rVqFPeE18vTx5um9zk5gYMwamTYOaNc0qvJtuKrGkbGEsH9IRQhRNaqrpXdeoYQL43r0mgOc3pHLqlOlZAyxaBL17myHiwYPNmHirVvDrr+b1F5tSWK2aWfkrCvb8/G38vuMEq5/qgZenB2eT06jklzk8ozX88IP5X/Pff+HBB83/ssVIyrr0kM7ixeb+mmusbYcQVsnIMDEg5/BJfuPgjz0Gt99uxrvbtjWB+8Ybzfj5Cy+Y1Zz2XnbLltC9e+7A3aaN+X4DBpjVoB6ZlbSuu87cRNEcjjvPjBUHeOraZgQFeHNd61q0uKwStsz/YLOC/ZEjZkrRzz+bf4QffoDLL7ekzZYHfHutEgn4oryw2UzPOm/AbtoUevUywya9e8N995m6Lvv3m2N5+fpmB/CaNc2wCpghlc8/z55d0qOHCeA+Po61r6DhFlGwxNR0UtNtVA7w4VxyOj/8fYx+4bW4snEInRoE06lBcPbJ6ekwdWp2UvaNN8z4l5d1YdfygC+EK0hMNLeQEPP8yy9NIO7VywT4a6/NvTIzI+PC97j7bnO+v7+52QN0aKgpSJZ3KKVChfyHditVMmPmdhbGD7eSlJpBl1f+5Jb2dXimX3PCawex4blrqOCbzz/Ahg2WJGULI78qwi2lp5sZKfYh1B9/zD21MO/t/HkzRPLnn+b8F180wyq9emUPjYSFZa/MzO8WnNn5Uwr++CO7LQEBcP/9ZfajiyJY+M+/7P7vHGOvaYK/jycP92xMmzqVs45fEOzPnoVnnzUBvlatMk/KFkYCvihXEhPNEMmBA6ZiYc7xcA8Pk5wE0+k6cwbWrzfPJ0wwnTH7ykz7rXHj7Mc5h11WrjQ9bbucAVy4tqOnEqlT1WSpow6fZuXeWB7s3ghvTw+Gd6mf/4u0NkmVMWOyk7KTJ+f+JXECEvCFS9q/39QwGTDAPB83Dj77zPyt5VWlignYOT9RP/BA7imIP/9s6pDntzIzPzVqXFr7hXP6ZetxRn+1iQVjrqLFZZV4vHdTnruuOaqgX4rDh01S9pdfLE/KFsbygD9jhtUtEM5Ga9Mj37fPBPac9wsWmKGRzz83C4GSkkxyMzTUzA1v1MgkNRs2hMsuM2Pu+SUzb7op9/PQ0LL52YRzSUhJZ/rSfVzRMJirGlfjykYhPHFtU2oG+QHg71NAhjs9Hd55xyRlweyGM2aMUydVLG9ZfrMTRPlns5ndkKpXBz8/MzY+fXp2YD9/PvtcDw/TO2/Y0FQ3DA6Ge+81RQTtf1v2QlpCFCY9w8a/8cnUqRqAr5cHP246RgVfL65qXI3KAT482L1R4W/y11/mY+LmzWYbrmnTnCIpWxjLA/7PP5v766+3th2i5KWlmfopOXvoQ4aYxOYff5ge+fLlcPXVZjx9+3bTQ+/a1dzbe+thYRf20uvUMTchimrE5xs5fiaJhQ9fhbenB38+3g0/bwfnqsbHm9oQ9qTsd9+ZxRBOkpQtjOUB374nqAR812SvjxIXZ/ZntQf2ffvMepOc0xMDAkzVw44dzf2MGSaogxliyTvMIkRJ2Bp9htmrDvLqoNb4enky7IowUtIysn53HQr2eZOyo0ebRUROlpQtjOUBXzi/+PjsXnq1amZ6YnKy6X0//DA8+aRJgD7+uEmQNmpkSt/ecUd2L71RI5PotHeEqlc305SFKA0xZ5Px8/Gkkp83pxPTWLUvjgOx52leqxJdm1Qr2pvlTMpGRDh1UrYwEvBFlnXrTF2WvMnSkyezzxk0yAR8Pz/zSbZFC/P12rVNL79qVWvaLoTdf2eT6fLKnzxxbVMe6NqQqxqFsGZcD3y8PIr2Ri6YlC2M67ZcFInNBseOmeAdGWm+9sQT5uv2YbVbbjGJVA8Ps8FEw4YmMWrvoTdsCA0aZL/ntGnZjz08JNgL63yy+iCJaRmM6taIGpX8eL5/i6yevIeHwsejiGPs69ebpOyWLWa8+d13XSIpWxgJ+OVIWpoZN7ePoeccTz9wwNRbqV3bBHUwwzI2W/brv/3WDMmEhZmpjkI4K601e2MSaFKjIgCbj57hXHI6WmuUUgy7Iqx4bxwfb1bKvveeSyZlC2N5eeSjR81zmXHhGPu8cw8PM9Nl0SJTkwlMydu5c7PP9ffPPYbesKFZOdqzpzVtF6KkzFyxn1cW7mLVUz24rLI/qem2og/Z5KS1Ce4PP+z0SVmXLo8sgf5C8fGmd5530dH+/aZ3fviwGXLZtAlmzzZ1XQIDzdz0vn2zA3zNmuWmYyLcXMzZZF5ftJuhnerRpk5l+rWqRWV/H6oEmPm6lxTsDx0yAf7XX8145/z50KFDyTTcyVge8L/5xtwPHmxtO6ygtRlqqVnTBOw5c0xOKDY293k1apgA3qOHubcPtzz2mJkhY9e7d9m1XYjSlpCSTlxCCvWCAwnw9WLZnlg6NQimTZ3KhFYJ4NYOl7grS1qaScqOH296Rm++aTYoceGkbGEs/8nef9/cl/eAb7OZHvrGjRAebm4rV5pFRr/+ajZyrl/f5IeaNs3upTdoYLaVy4/UNRfl2aD31xBcwYcv7+tEBV8v1ozrgbfnJfTkc8qblJ02zXxsLucsD/jlkc1mpjdu3Ghuf/9tbmfPmuPPP28Cvn3xkX0noo4dzU0Id7R0dwzfbYzm3SGRKKV44tqmVAnMXmJdIsE+Z1L2ssvMYqqBA91m7FMC/iXKyDCBvEoVM+uld28ztp6QYI77+pqAfscd0K6dqaHesqU5VrGiLD4S7m3vf+eoUzUAP29PTp9PZdeJc1kbgfdsXoIlSe1J2TFjTN3shx6CiROdMilbmiTgF0F6OuzaZeqr22e6tG1rZr7Mm2cWIwUHm23r2rY1Ab55c/D2trTZQjilf6LjuX7aKt64pQ2D2oUyIKI2N0bWLrgUcXHkTcr++GO5TcoWRgL+RaSlwc6d2cMyGzea4b6kJPNJ8Ngxc94jj5ga6nY//GBNe4Vwdjab5tVFu6hZyY+7u9QnvHYlJg0Mp3tTs0DKs6iLowqTlgZvv212t3GTpGxhLP/J582zugXm98LeC//gA/j4Y9i61QzRgNlbtG1bk+OxD8vYCy8NH25Zs4VwemkZNnb9e45WoUF4eCj2nDhHSppZ7aeUYminUlq9un69GS/dutWtkrKFsTzg2zeFtsrs2WY3shMnzP6mCQmmquODD2YH98aNs/ctFUI4bvKvO/lmw1HWP9uTSn7efDSsQ8n35HOKj4dnnjHT/9wwKVsYywP+J5+Y+7LoKaenmx2TZs2CsWNNEbBWrcynvNRUc87jj5ubEKLo9sUk8PKCnYy/vgX1ggMZ2qkuVzcJIdDHhJpSC/Zam+GChx82SbYxY0xS9mJzmt2UWwT8vXtNT/7TT82q6Ro1zO8EmNyNm+ZvhCgRJ+KTSU23UTc4gAq+Xuz69yxHTiVSLziQRtUr0qh6KQfdQ4fMR/IFC8xH8p9+gvbFqjxQ7lke8EtLYqKZhfXRR7BihRmSue46U36gXz+ZOSNESUjLsNFv6kq6NArh3SGR1AzyY9VTPfAozWGbrG+emZQdP978gb/1lpmN48ZJ2cI4NDKtlOqjlNqtlNqnlBqXz/G6SqmlSqlNSqmtSql+Jd9Ux02caArd3XUXHD8OL78MR4+a//gHDJBgL8SlmBt1lIfnbALMYqhXb27N472bZB0vk2C/bp3pxT/5JPTqZabUjR0rwb4QhQZ8pZQnMB3oC7QAhiilWuQ57TlgrtY6ErgNeK+kG1qQ06fNwjn7OLy/vwnsy5bBnj3w9NMmfyOEKDqtNX8dPEWGzVTWjU9M40R8MkmpZv/KXi1qUC84sGwaEx9vhm+uuMLsuPPDD2ZevVRhdIgj/x1eDuzTWh8AUErNAQYAO3KcowH7krUg4HhJNrIw69eb34EGDczG2JJ0FaLkLN4Zw/2fRfHx8A50b1ad+66qz/1XNyj8hSXJnpQdMwZiYiQpW0yOBPzawNEcz6OBvBVfJgC/K6UeAgKBaxxtwIIFjp55oZQUU7qgVy/YvDm7Jo0QoviSUjN4dv4/dG4QzC3t69C1STXeGtyGzg2DAUp+JWxhDh40PbqFC01S9uefJSlbTI6M4ef3r5t315QhwCda61CgH/C5UuqC91ZKjVBKRSmlomIzawAHBJhbUe3cCU2awO+/m6qREuyFKL5zyWlsOnIaAD9vD6JPJXE60YyR+nh5cGNkKH7eZVyeNS0NXnvNFJ9ascIkZdevl2B/CRzp4UcDOQfIQrlwyOZeoA+A1nqtUsoPCAFicp6ktZ4JzASz4xWYsXeAUaMcb/SBA3DNNaZwWf36jr9OCJG/J+dtZcOh06x92pQg/uaBTmXfk89p3TqztH3rVpOQe/ddGacvAY708DcAjZVS9ZVSPpik7E95zjkC9ARQSjUH/IA823jkb+7c3NvyFebYMRPsk5PNFn+NGzv+WiGEEXXoFIPeX8Pp86YXP7pHI2YPb59VgtiyYH/mjOn9XXEFnDplkrLz50uwLyGF9vC11ulKqdHAIsATmK213q6UegmI0lr/BDwGfKiUegQz3DNcl8JmuTExJtifPAlLlphVskIIx+w6cZaKft7UruxPoK8X55LTOR6fRJVAH1peFmRt47SGb781K2VjYsz9Sy9JUraEOTRpVWu9AFiQ52sv5Hi8A+hSsk3L7fRpU2v+8GGzcbesjhXCcfGJaVz/7iru7BTGC9e3oHmtSvw29iprh23s8iZlf/nFFLISJc4lVimcO2dWx+7caRZPXXWV1S0Swvm9u2Qv0aeTeHVQa4ICvPlgaDva1q2SddzyYJ+WZhKxEyaYmRdvv20CvyyeKjVOXwMyLc3kbDZsMJt8X3ut1S0SwjmlpttYujt7nkRKuo3k9AxsmQumejavkWvLQEutXWt68U89Zf6od+wwwzgS7EuVKoWhdoe0b99eR0VFFXre+fPmk56vrylrLYTI3xfrDvPc/G388tCVhNe2eEz+Ys6cMeWLP/gAatc2s28GDrS6VS5FKbVRa12sualO/99pYCAMGmR1K4RwPrHnUnjs2y0M7ViX3i1rckPEZYRW8adFLSfcp1VrMx1v7FhJylrI8iGdN94wt4v57Tf4+++ya48Qzuz4mST+zlwgVSXAm3PJaSSlmZo2lfy86da0etkULyuKgwdNEu6220xRq7/+MmP3EuzLnOVDOt26mefLluV/XpMmZvrld9+VVcuEcF43vbeahJR0Fo292vqka2HS0sw+si++aJKykyZJUrYElOshnT//NBuHC+GOftt2gulL9/Ht/zrj5+3JhBtaUiXAx/mD/dq1ZqXsP//AjTfC1KkQGmp1q9ye5UM6hQkNldW0wn3YbJrV+05yMiEFgIp+XlT08yIuc0Vs69DK1KlajOJTZeXMGRg5Erp0MYtn5s83+8pKsHcKTh3wf/8d/u//zF60QriDI6cSueOj9czbGA1Al0YhfHV/J2pX9re4ZYXQGr75Bpo1g5kzTXJ2xw4zp1o4DcsDvr+/ueXn669NsTzPMi7SJ0RZGvfdVl78eTsAYSGBfHrP5Qy/IszaRhVFzqRsaKhZNPPmm5KUdUKWB/yFC80tP5s2QWQkOPtwpRBFEZ+YxqLtJ7Ke+/t45io93LVJtbIvRVwcaWnw6qumfPGqVfDOO6Z8cdu2VrdMXITTJm1TUmD7dujb1+qWCHHp7LPhlFLMWn2Qd//cy9pxPakZ5Mf461ta3LpiWLsWRoyAbdskKetCLO/hT5xobnlt327G7iMjy75NQpSkfTEJXDd1Vdb8+aGd6vLz6CupGeRnccuK4fRp+N//TPni+Hizn6wkZV2G5QF/yRJzy2vTJnMvAV+4om3H4tl89AwAtYL8CPT1JDnNBkD1in7OW/rgYrQ2xayaN4cPP4RHHzVJ2RtusLplogicdkhn0yaT82nY0OqWCOEYrTVKKWw2zagv/6ZecACf39uRQF8vvv3fFVY3r/gOHDCbkixaZLYXXLBAxuldlOU9/Iv5+2+zT62H07ZQiGyfrz3EDdNWY7NpPDwU026PZNoQFw+KaWnwyismKbt6tRmnX7dOgr0Lc8pwmpEBW7bIcI5wXinpGfy4+RiJqWaRSNVAX+qHBHIuxTxvHVqZoABvK5t4adasMYH96aezN6N46CGZI+3iLA/4wcHmltPx46bchgR84Wzss23+iY7n4Tmbs6ZXXte6FlOHRBLk78JBHrKTsl26ZCdlv/tOkrLlhOXF0y7GZjM9fW8X//sR5UNaho17PtlA27pVeKRXE7TW/HXwFB3CqjpfdcrisK+UHTsWYmPN/YsvQoUKVrdM5HEpxdMs7+FfjIeHBHthraOnEln4z78AeHt6EFrFn5AKZscopRQdGwSXj2B/4IBZ8DJkCNSpY1bK/t//SbAvhyyfpfP00+Z+ypTsr40eDTVrwnPPWdMm4b7sSVeA95fv58dNx+jWtDr+Pp5Muam1xa0rYWlpJrC/+KLpXU2dambjyDh9uWV5D3/tWnPLKS7OFN0Toiz9dfAUV776JwdiEwAY3b0Rvz/aFX+fchgAV682STJ7UnbHDknKugHLe/j5+fprq1sg3IHNplm57yQhFXxoeVkQYcEBNK5RMWuB1GXOXqGyOE6fhnHjTEXLunXhp59ks2g3YnkPPy+LcsjCjWTYzC9ZaoaNh+ds4pPVhwCoXsmPT++5nBaXOeGesJdKa9OTat4cPvoIHnvM1C+RYO9WnC7gT55stjRMS7O6JaI8evW3XQyZuQ4AP29PvryvI5NuDLe4VaVs/37o0wduv9306qOizEbSkpR1O5YH/NDQ3FN8o6JMsJcZOqIknElM5fO1h0jPMMM0YcEBhNcOIi3zecvLgvD1Kqfj1qmpZjZEeLhJlL37rrmXBS5uy/Ix/C++yP182zZo186atojyQWtNhk3j5enBugNxPP/jdhpVr0jnhsEM7lDX6uaVjdWrzZ6y27fDzTebWvW1a1vdKmExy3v4ecXGQq1aVrdCuKr4xDT6vrOSr/46AkDP5jVY+PBVdG4YXMgry4nTp02gv/JKOHfOJGXnzZNgLwAnCPhjx5obmKGcs2ehalVr2yRcy64TZ/ltm1kgFRTgTXjtIKpXNLXmvT09aF6rHCZh87InZZs1g1mzJCkr8mX5kM7mzdmPT5v9IS6orSNEQV77bTd7/jtH7xY18fBQvHFLG6ubVLb27zcLpn7/HTp0gN9+k3F6kS/LA35Op06Ze+nhi4KcSUzlg+UHGHF1A6oG+jBpYDiBPl7lo8xBUaSmmtk2EyeaWQ7vvgsjR8riKXFRlg/p5BQXZ+4l4IuCxJ5LYdaqA6zcGwuYBVIuXYq4OFatMr34Z5+F/v1N+eLRoyXYiwI5VcC39/BlSEfkNW9jNG8v3gNA4xoVWT2uBwMi3DARefq02Tz8qqsgIQF+/hm+/VaSssIhDgV8pVQfpdRupdQ+pdS4i5xzq1Jqh1Jqu1LqK0cb0KSJuYHp2Q8cKLN0xIW2HD3D6n0ns+bT25OybkNr+Oork5SdPRsef9wkZfv3t7plwoUUWg9fKeUJ7AF6AdHABmCI1npHjnMaA3OBHlrr00qp6lrrmILet7B6+MK9HTx5nqe/38rEAeGZ9W0y8PXyQCk3G6cHk5QdORL++AMuvxxmzICICKtbJSxS2vXwLwf2aa0PaK1TgTnAgDzn3A9M11qfBigs2AtxMbbMOjdB/t7EnE3h2JkkwJRBcLtgn5oKL79sVsquWwfTppmtByXYi2JyJODXBo7meB6d+bWcmgBNlFKrlVLrlFJ9HG3AiBHmBjBmjKntJNzTKwt3MeLzjQBUDfRh8aNd6da0usWtskjepOyuXfDgg5KUFZfEkWmZ+XWr8o4DeQGNgW5AKLBSKRWutc5V1V4pNQIYAVC3rlnivmdP9vErroDKlR1ruCgfElPTCfAxv4bVKvqSlJpOeoYNL08P95tmCWbmwrhx8OGHptDZzz/LOL0oMY4E/GigTo7nocDxfM5Zp7VOAw4qpXZj/gPYkPMkrfVMYCaYMfy83+i22xxvuHB924/Hc+esv3h7cARXN6nGvVfWt7pJ1rGvlH3kETM/+fHHYcIECAy0umWiHHFkSGcD0FgpVV8p5QPcBvyU55z5QHcApVQIZojnQFEbEx9vNi4X5ZfNpok5mwxAo+oV6NqkGiEVfC1ulcX27YPeveGOOyAszJSMff11CfaixBUa8LXW6cBoYBGwE5irtd6ulHpJKXVD5mmLgDil1A5gKfCE1jquqI2JiIC77y7qq4QrGTNnE3fO+osMm8bXy5O3BkeUzw1HHJGaajaACA+Hv/6C6dMlKStKlUOlFbTWC4AFeb72Qo7HGng081YkOX+34+Jk0VV5tPvEORpWC8TL04Ob24Zy6nxqvokht7JqlalquWMH3HILvP02XHaZ1a0S5ZzltXTeftvcp6WZaq5SVqF8+Sc6nhumr+LlG1sx5PK6dG/mprNu7E6dgqeeMtsM1qsHv/wC111ndauEm3Ca0gr2SpkS8F3f6fOpRB0ydTLCa1diwvUt6Rfu5suntYYvvzQrZT/+GJ54wqyUlWAvypDlAX/oUHOTwmnlxxPztjDyy79JTbehlGLYFWHuV9wsJ3tSduhQaNAANm6E116TpKwoc5YP6URHm3spnOa60jNsfL/pGH3Ca1LJz5sn+zRDa/Dxsrw/Ya2W/HCPAAAgAElEQVTUVDPbZuJE8PU1SdkHHpDFU8Iylgd8O6mF77r2xiTw5LytJKakM7xLfZrUqGh1k6y3cqUJ7jt3SlJWOA2n6YJJwHctfx85zdeZ+8Y2r1WJH0ZdwbArwqxtlDM4dQruvx+uvhoSE01Sdu5cCfbCKThNwJcxfNfy1fojTPtzHynpZqVcZN0q7lfcLCet4YsvJCkrnJrlQzqdO5v7tm3NqvJKbroGx9nFJaTw9uK93H9VA+oGB/BMv+b4enng6yXj0ezda8oXL1kCHTuaMsZt3GxfXeESLA/4U6ZkP+7WzbJmiEKk2zTzNx+jTZ3K1A0OoGqgj9VNsl5qqpltM2mSScq+954p/SpJWeGkLA/4dqdOmb8ZmanmPOb8dYR/jsUz+cZW1Kjkx5pxPajo58bTK3PKmZS99VZ46y0ZpxdOz/KAf/PN5v7sWTh/3pQSEdbRWmeNxf8bn8yB2PMkp2Xg5+0pwR5Mz+TJJ2HWLLNS9tdfoV8/q1slhEMsD/j2ZO3YsZCebm1b3N3huPM89PUmXujfgvZhVXmoRyM8PZR7J2Pt7CtlH300O+i/8IJ8JBUuxfKAbzdwoNUtcF8p6Rn4enlSraIvnh6KhBTzP6+Xp9NM4rJW3qTs4sXQurXVrRKiyJzmL3r7dvj3X6tb4X5eWbiLwTPWobUmwMeLH0Z1cd9tBfNKSTEJ2VatYMMGk5Rds0aCvXBZThPwu3c3K9BF6TuTmJq1WXjzWhXp2KAqqRk2i1vlZFasMHvKPv88DBhg9pQdORI8nOZPRogis/y3t2dPc0tMhIAAq1tT/u2LOcdVry3lxy3HABgQUZun+zaX+fR2cXFw773QtSskJcGCBfDNN1DLzat9inLB8oD//PPw3HPmb8vf3+rWlE8ZNs2hk+cBaBBSgVva1aHlZUEWt8rJaA2ff25Wyn76qalZv3079O1rdcuEKDGWB3ww61dsNgn4peXxb7cw5MN1JKdl4OGheOH6FlLgLKe9e6FXL7jrLmjUCP7+G155RT5yinLH8lk6ffua3a5A/r5K0qYjp2lasyIBPl4M7VSPXi1q4Ovu5YrzSkkxK2UnTwY/P3j/fbNSVsbpRTll+W92UpIZvwfp4ZeU/bEJ3PjeGj5bexiAdvWq0K9VLZlPn9OKFWZD5RdeMHOCd+6E//1Pgr0o15zitzvDFFyUHv4liDmXzJ+7/gOgYbUKTB0SyZ2d6lncKieUMymbnGySsnPmSFJWuAWnCPi2zBmB0sMvvikLdjF2zmaSUs3/nje0uYxAX8tH7JxHzqTsZ59JUla4JQn4Lio13canaw5xIj4ZgMd6N+Gn0Vfi7yPTKy+wZw9cc40kZYXbs7wL2L8/nDtnPmW3amV1a1zHf2eTmfzrTpLTMniga0NCq0jwuoAkZYXIRWmtLfnG7du311FRUZZ8b1e1dn8cfx85zYPdGwGwLyaBhtUCJRmbn+XLTfni3bth8GBTvljG6UU5oJTaqLVuX5zXOkVXJy4OtmwxHTJxcct2x/DV+iOczyxu1qh6BQn2ecXFwT33mN10UlNh4UJJygqRyfKA360bXHmlmSF35IjVrXEucQkpPDp3M/9ExwMwpmdjljzWVZKx+dHaJGObNTPJ2XHjYNs26NPH6pYJ4TQsD/gAlSvD99/LhkF5eXt5sHZ/HDtPnAUg0NcLP29Jyl7AnpQdNgwaNzZJ2SlTJCkrRB5OEfB9feHGG2UvCYAv1x/mvk+j0FpTyc+bZU9049b2daxulnNKSYGXXjLZ/o0b4YMPYNUqyf4LcRFOEfATE81Qq81NK/Rm2DT25LnNprFpTWLmfHqpYnkRy5dDmzYwfjzcdJMpX/zAAzIDR4gCOMVfx3//wXXXgTvmH4+dSaLvOytYtN2skh3aqR6zh3eQcfqLyS8p+/XXULOm1S0TwulZHvBvvdUMu/r7u1fAP5dsKsbVqOhL3aoBWQumZNbNRUhSVohL5lDAV0r1UUrtVkrtU0qNK+C8QUoprZRyeI7oqFHQtKl7rbJ99bddXP/uKlLTbXh5evDRsA50bVLN6mY5rz17zC45kpQV4pIUOm6glPIEpgO9gGhgg1LqJ631jjznVQTGAOuL0oDERDh7tvz/7cacSybI3xtfL086NwjGz8sTm0WL3lxGSoopgfDyy6ZH8MEHcP/9Mk4vRDE58pdzObBPa31Aa50KzAEG5HPeROA1ILkoDejXDxYtKt89/KOnEun62jI+zyxXfHWTajx8TWOZYlmQZctMUnbCBLj5ZknKClECHPnrqQ0czfE8OvNrWZRSkUAdrfUvxWmEzVb+evip6Ta2HTMLpupUDWB0j0b0bF7D4la5gJMn4e67za72qanw22/w1VeSlBWiBDgS8PPLImaNRSilPIC3gMcKfSOlRiilopRSUbGxsVlfz8gofz388T9tY8iH6zibmZx9sHsj6ofIQoOL0trsJdusGXzxBTz9tEnKXnut1S0TotxwJOBHAzlX/oQCx3M8rwiEA8uUUoeATsBP+SVutdYztdbttdbtq1XLTlKWlx7+mv0niUswBYHu6VKfqbdFUlGmVxZu927o0QOGDzcZ/E2bzLh9efilEMKJOBLwNwCNlVL1lVI+wG3AT/aDWut4rXWI1jpMax0GrANu0Fo7XAqzPGxgfuxMEnfO+ovZqw8C0LhGRbo3qy7TLAuSkgIvvgitW8PmzTBjBqxcCeHhVrdMiHKp0O6n1jpdKTUaWAR4ArO11tuVUi8BUVrrnwp+h4INHw6dO5ttRV1N9OlE/jp4ipvahlK7sj8fD+/A5fWrWt0s17BsmUnC7tkDQ4bAm2/KOL0QpUzq4V+C8T9uY97GaNaM60lQgLfVzXENJ0/CE0/AJ59A/fpmUxIZpxfCYS5dD//kSbMyfseOws+1WnJaBu8v28/e/84B8PA1Tfjj0a4S7B0hSVkhLGd5wB80CIYOhVmzrG5J4RJTM3hv6T4WbT8BQNVAHy6r7OLJh7IgSVkhnILlAR8gMhLGjrW6FflbsSeWCT9tB0yA/+PRrozu0djiVrmIlBSzcMqelJ05U5KyQljIKeYMVqgAdZy05PuuE2dZtjuGM4mNqRzgQ80gP6ub5BpyJmVvv90kZWvIwjMhrOQUPfzjx82nfGeSlmGK8w+/oj5/PNqVygE+FrfIRZw8aYZuuneH9HRTN+PLLyXYC+EELA/4WsPevaasubP46+Aper25nH0xCfh4eeDtafllcn5am5k3zZqZAP/MMyYp27u31S0TQmSyPJLde6+593SiOmIV/by4rLI/wYHSq3fIrl2mR3/33Sbgb94Mkye7/mo6IcoZywP+jTeaey8nyCakppthnOa1KvHV/Z2oIgG/YMnJJinbpg1s2WKSsitWQMuWVrdMCJEPywP+oUPm3uoe/rnkNG58bzWzVx20tiGuYulSE+hffBFuucX08qVWvRBOzfK/zv/9z9xb3cP38fKgSY2KNKxewdqGODt7UrZHD1Pm9PffzUIqScoK4fQsH0ixV3awKuBrrUlJt+Hn7clbgyOsaYQrsCdlH3/cbFH2zDPw3HMyTi8KlJaWRnR0NMnJRdoXSQB+fn6Ehobi7V1yK/mdJuBbNaTz+qLdrDsQxxf3dSTAx/LL4Zx27TIfxZYvhy5dTFVLGacXDoiOjqZixYqEhYVJ5dgi0FoTFxdHdHQ09evXL7H3tXxIx+oefqvaQbSpUxl/2W7wQsnJMH58dlL2ww8lKSuKJDk5meDgYAn2RaSUIjg4uMQ/GVnepbUq4J9PSSfQ14u+rWrRt1Wtsv3mruDPP02vfu9euOMO+L//k3F6USwS7IunNK6b5T38J580q+779Su77/n3kdNc9dpS1h+IK7tv6ipiY2HYMOjZ0+xMI0lZ4eI8PT2JiIggPDyc66+/njNnzgBw6NAh/P39iYiIyLqlpqZa3NrSZXkP/+aby/57hlbxp3ODYBrJjJxsOZOy587Bs8+amyRlhYvz9/dn8+bNAAwbNozp06fz7LPPAtCwYcOsY+7A8h7+unVmKrd9Pn5pOpechtaa6hX9mH5HW4Ir+Jb+N3UFu3ZBt25wzz3QooUpbDRpkgR7Ue507tyZY8eOWd0My1jewx89GjZuhIgICAsrve+TlJrBbTPX0SGsKhNukKQjYJKyU6aYW4UK8NFHpjyCLJ4SpaRbt8LP6d/ffNC0nz98uLmdPGn2z8hp2TLHv3dGRgZLlizhXns9F2D//v1ERJjp2F26dGH69OmOv6ELsjzgBwaaPW1Lu8aWn7cH1zSvQUTdyqX7jVxF3qTsm29C9epWt0qIEpeUlERERASHDh2iXbt29OrVK+uYuw3pWB7wlQIfn9IbPdBaE5+URuUAHx7p1aR0vokriY2Fxx6Dzz+Hhg1NUjbHH4AQpakoPfK854eEFP31kD2GHx8fT//+/Zk+fTpjxowp+huVA5Z/dk9ONuP3B0uphM07S/bS/91VxCWklM43cBVaw+zZpprlnDlmlew//0iwF24jKCiIqVOn8sYbb5CWlmZ1cyzhFAH/8GFzKw09mlXnuta1qOrOlS937jSDoffea5KymzfDxImSlBVuJzIykjZt2jBnzhyrm2IJy4d0br/dLOIs6YVXJxNSCKngS+vQyrQOddNx++Rks1n4K69IUla4rYSEhFzPf/7556zH27ZtK+vmWMryv/zWrc19SdbS2XYsnqtfW8rPW46X3Ju6miVLzMWdOBEGDzZTL++9V4K9EG7M8r/+vXvNfUn28BtWq8CgdqF0bhhccm/qKmJj4a674JprzLj9H3+YBK3MwBHC7Vke8GfMMPclEfBPn08lJT0Dfx9PXhoQTog7LazKLym7dasJ/EIIgRME/JIqj5yeYWP4x38x8ou/0fY3dRc5k7ItW5qkiCRlhRB5WJ60tbvUHr6Xpwd3d6lPRT8v96nOJ0lZIUQRWB7wL7U8staa4/HJ1K7sz8DI2iXXMGe3eDGMHAn79sHQoaZ8sYzTCyEKYHlX0B7wi9sp/WjlQfq8tYLDcedLrlHOLDYW7rwze8GUJGWFuKhDhw4RHh6e62vLli1DKZVremb//v1ZlrmMt1u3brRv3z7rWFRUFN0cKQLkAiwP+NOnw/z5xS+cdl3rWtx7VX3qVg0o0XY5HZsNZs0ySdlvvoHnnzcrZSUpK0SRhYaGMnny5Isej4mJYeHChWXYorJhecDv0QMGDCj6kM7huPNorbmssj9jr2lSvsftd+wwSdn77stOyr70Evj5Wd0yIVzGgQMHiIyMZMOGDbRp04agoCD++OOPfM994oknmDRpUhm3sPRZHvA/+8xMLjl1yvHXHIhNoM/bK5m1qpQK8DiLpCTTk4+IgG3bTA9/2TJo3tzqlglRLINnrOXbqKMApGXYGDxjLT9sigZMCfPBM9ZmLZg8m5zG4Blr+W3bvwCcOp/K4BlrWbzjPwBizjm+3+vu3bu5+eab+fjjj+nQoQMAzz333EWDeufOnfH19WXp0qXF+0GdlEMBXynVRym1Wym1Tyk1Lp/jjyqldiiltiqlliil6jnagClTzPTxogT8sOBARvdoxICIcpykXbzYrJSdNAluu82slL3nHpmBI0QRxcbGMmDAAL744ous2vcAV111FQArV67M93UF/YfgqgodSFFKeQLTgV5ANLBBKfWT1npHjtM2Ae211olKqZHAa8BgRxpQo4a5NWpU+LlxCSkopaga6MOD3R14gSuKiTHli7/4wlyUxYvN/rJClAPfPNA567G3p0eu5/4+nrmeV/LzzvW8aqBPrufVKzo2pBkUFESdOnVYvXo1LVvm3vzo2WefZfLkyXjlM6bco0cPnn/+edatW+fQ93EFjnQXLwf2aa0PaK1TgTnAgJwnaK2Xaq0TM5+uA0JLtplm+uWoL//mjo/Wk2Erhwur8iZlX3jBJGUl2AtxSXx8fJg/fz6fffYZX331Va5jvXv35vTp02zZsiXf1z777LO89tprZdHMMuFIwK8NHM3xPDrzaxdzL5BvelspNUIpFaWUioqNjQXg7FnYs8d0bAuilOKJa5vyWK8meHqUswRtzqRsq1YmKfvii5KUFaKEBAYG8ssvv/DWW28RHx+f69izzz5LdHR0vq/r168f1apVK4smlglVWBkCpdQtwLVa6/syn98JXK61fiifc4cCo4GuWusCdxxp3769joqKonlzMzy9d2/+wzpaa3adOEfzWpUc/qFcRlISTJ4Mr70GFSvC66+bzTtlnF6UEzt37qS5TDIotvyun1Jqo9a6/UVeUiBHIks0UCfH81DggrrDSqlrgGeBGwoL9jkNHVrw8a//Okr/d1ex5egZR9/SNSxebHrzkyfDkCGSlBVClDpHZr9vABorpeoDx4DbgNtznqCUigRmAH201oUMzuRWr5D5PDdEXEZSWgatQ4OK8rbOKyYGHn0UvvwSGjc2det79LC6VUIIN1Bod1JrnY4ZplkE7ATmaq23K6VeUkrdkHna60AF4Ful1Gal1E+ONuBiG87Yh5oq+Hpx75X1XX9hlc1mips1awZz55qk7NatEuyFEGXGofWtWusFwII8X3shx+Nir+//8cf8v/7d38eYsXw/X97f0eHpV05rxw544AFYtQquvtpsAtCsmdWtEkK4GacdMA4O9KFR9QqEBLrwJiZJSWYjkogIE/RnzzYrZSXYCyEsYHl55Ivp3qw63Zu5cAXIP/4w5Yv37zdbDr7xBpSj6V1ClDerV6/GZrNlrcAtj5yyh5+abiM13WZ1M4onJsZMPerd28y4WbIEPv1Ugr0QTiQsLIyTJ09mPd+0aRMff/wxnTp1KvS1b7/9NomJiYWe169fP86cca7ZhU4Z8P/cFUOrCYvYdeKs1U1xnCRlhXBZkZGRfPTRR3h7exd6rqMBf8GCBVSuXDnX17TW2GzWdWYtD/jz55sp6DkXXdWtGsDQTvWoHxJoXcOKYvt2k4y9/34zt37rVlkpK4STGDhwIO3ataNly5bMnDnzguNffPEFl19+ORERETzwwANkZGQAMHLkSNq3b0/Lli0ZP348AFOnTuX48eN0796d7t27A/D111/TqlUrwsPDeeqpp7Le1/4p4tChQzRv3pxRo0bRtm1bjh49yu+//07nzp1p27Ytt9xyCwkJCWVwJRxYaVta7CttXVpSkqlm+dprUKmS2WZw2DBw9SmkQpSQXCtFx46FzZtL9htERMDbbxd4yqlTp6hatSpJSUl06NCB5cuX065dO6KiooiNjeXJJ5/k+++/x9vbm1GjRtGpUyfuuuuurNdlZGTQs2dPpk6dSuvWrQkLCyMqKoqQkBCOHz9Op06d2LhxI1WqVKF3796MGTOGgQMHZp2XkJBAgwYNWLNmDZ06deLkyZPcdNNNLFy4kMDAQF599VVSUlJ44YUXLmh7Sa+0tTxp+/rrZtGpfZe+5LQM/jubTN2qAc499z5nUnbYMPODyDi9EE5n6tSp/PDDDwAcPXqUvXv3Zh1bsmQJGzduzKqRn5SURPXM7ULnzp3LzJkzSU9P599//2XHjh20bt0613tv2LCBbt26ZdXbueOOO1ixYgUDBw7MdV69evWy8gPr1q1jx44ddOnSBYDU1FQ6d+5MWbA84H/yidmH+3zmlrR/Hz7N7R+t59N7LqdrEycMoP/9Z1bKfvUVNGkCf/4JmR/thBAFKKQnXhqWLVvG4sWLWbt2LQEBAXTr1o3k5OyNU7TWDBs2jClTpuR63cGDB3njjTfYsGEDVapUYfjw4blel/P1jggMzB6e1lrTq1cvvv7662L+VMVn+Rh+tWrQuTPUr2+eN6pegUkDw4msW7ngF5Y1mw0+/NAkZefNg/HjTVVLCfZCOK34+HiqVKlCQEAAu3btuqC2fc+ePZk3bx4xmeV6T506xeHDhzl79iyBgYEEBQXx33//5drftmLFipw7dw6Ajh07snz5ck6ePElGRgZff/01Xbt2LbBNnTp1YvXq1ezbtw+AxMRE9uzZU5I/9kVZ3sPPq3olP4Z2cnjDrLKxfbtZKbt6NXTtCh98IIunhHABffr04YMPPqB169Y0bdr0gmmXLVq0YNKkSfTu3RubzYa3tzfTp0+nU6dOREZG0rJlSxo0aJA1/AIwYsQI+vbtS61atVi6dClTpkyhe/fuaK3p168fAwYMyNuMXKpVq8Ynn3zCkCFDSEkxdSYnTZpEkyZNSv4C5GF50jYiAo4eNXt9VK9hY+W+k7SvV4WKfoVPjyp1OZOyQUFm8ZQkZYVwmJRHvjRWlEcuVSkpZj/bxETY+e857v54A0t3x1rdLPj9dwgPh5dfNgupdu0yteol2AshXJTlAf9//8t+3LhGBb66vyNXNw6xrkH//Qd33AHXXgteXiYp+/HHEGJhm4QQogRYHvCDg7Mf+3l7ckXDECoH+JR9Q2w2mDkzOyk7YYJZQCVJWSFEOWF5wLcnzbXWfLb2EIdOni/7RmzbBlddZRKzEREm0I8fD74uXKlTCCHysDzgL1li7k8kJPHCj9tZvf9kwS8oSUlJ8MwzEBkJu3ebRQF//glNm5ZdG4QQoow4zbTMWhUDWPd0T/y9PcvmGy5aBKNGwYEDJhn7+usyTi+EKNcs7+HnVDPIj6CAUp6OeeIE3H479OljkrJLl0pSVgiRr9jYWDp27EhkZCQrV6686Hn33XcfO3bsAODll18uq+YVmdME/E827mbt/rjS+wb2pGzz5vDdd9lJ2W7dSu97CiFcVnp6OkuWLKFZs2Zs2rSpwI1RPvroI1q0aAFIwC+4AR7g46v5YfshtkSX0mYBkpQVwi0dOnSIZs2aMWzYMFq3bs2gQYNITEzkpZdeokOHDoSHhzNixIismjjdunXjmWeeoWvXrrzzzjs8+eSTLFiwgIiICJKSkvItmWx/XVRUFOPGjSMpKYmIiAjuuOMOq37si7J8DH/5cgBFlaq9Scso4Y0BEhPNStnXXzcrZT/9FO68UxZPCWGR/D5Q33qrSaclJkK/fhceHz7c3E6ehEGDch9btqzw77l7925mzZpFly5duOeee3jvvfcYPXp0VjniO++8k19++YXrr78egDNnzrDcBCaCg4OJiopi2rRpAEyePDlXyeStW7fmqqD5yiuvMG3aNDaXdBnoEmJ5Dz8kxNw8PRR+JZmwXbTIbEYyZYoJ8rt2mb1lJdgL4Vbq1KmTVQtn6NChrFq1iqVLl9KxY0datWrFn3/+yfbt27POHzx48EXfa+7cubRt25bIyEi2b9+eNW7vKizv4T/1FMxffoabHjnOy7c2v/Qa+CdOwCOPwJw5Znrl0qUyTi+EkyioRx4QUPDxkBDHevR55Y0pSilGjRpFVFQUderUYcKECblKH+csZZyToyWTnZnlPfyFC+HoAR+W7TtxacHeZoMZM8xK2e+/N1sMbtkiwV4IN3fkyBHWrl0LmO0Ir7zySgBCQkJISEhg3rx5Dr1PQSWTc/L29iYtLa1kGl/CLO/hV60Kl1cNYOkzl1DCYNs2k5Bds8aUQnj/fVk8JYQAoHnz5nz66ac88MADNG7cmJEjR3L69GlatWpFWFhY1m5XhWnTps1FSybnNGLECFq3bk3btm358ssvS/JHuWSWl0e2d8CL81GNxESYONGULQ4KgjfflKSsEE7E6vLIhw4don///mzbts2yNlyKclceOS5O89euc3y1/ETRXvjbb6Z88SuvSFJWCCEcYHnAT0uHDJXB8TMOJj9OnIAhQ6BvX/DxMUnZ2bNlpawQ4gJhYWEu27svDZYH/CceV6SeqMzAlmEFnyhJWSGEuCSWJ20dWuz6zz8mKbt2LfToYZKyZbD/oxDi0mmtL326tRsqjfyq5T38b35NILD1EeIS8xnSSUyEp5+Gtm1h71747DNYvFiCvRAuws/Pj7i4uFIJXuWZ1pq4uDj8/PxK9H0t7+Fv3ehD2rlKpNvy/EL89ptZb33wINxzj9lIPOf2WEIIpxcaGkp0dDSxsU6wT7WL8fPzIzQ0tETf06GAr5TqA7wDeAIfaa1fyXPcF/gMaAfEAYO11occee8ATx9ST/hQo0LmF06cgLFj4ZtvzHj9smXQtatjP40Qwql4e3tTv359q5shMhU6pKOU8gSmA32BFsAQpVSLPKfdC5zWWjcC3gJeLXJLbDb44AMT5OfPh5degs2bJdgLIUQJcaSHfzmwT2t9AEApNQcYAOSsGjQAmJD5eB4wTSmltAMDd8kZaQRVPkWNWwbA1vWSlBVCiFLiSMCvDRzN8Twa6Hixc7TW6UqpeCAYKHSD2goJMdQ9sw8/td8kZYcOlcVTQghRChwJ+PlF37w9d0fOQSk1AhiR+TRBKbU783GIz2lOctddZrWsewvBgf8o3YBcB0OuQza5FkaxC4U5EvCjgTo5nocCxy9yTrRSygsIAk7lfSOt9UxgZt6vK6WiilsboryRa2HIdTDkOmSTa2EopaKK+1pH5uFvABorpeorpXyA24Cf8pzzEzAs8/Eg4E9Hxu+FEEKUnUJ7+Jlj8qOBRZhpmbO11tuVUi8BUVrrn4BZwOdKqX2Ynv1tpdloIYQQRefQPHyt9QJgQZ6vvZDjcTJwyyW044JhHjcm18KQ62DIdcgm18Io9nWwrB6+EEKIsmV5LR0hhBBlo0wDvlKqj1Jqt1Jqn1JqXD7HfZVS32QeX6+UCivL9pUVB67Do0qpHUqprUqpJUqpela0sywUdi1ynDdIKaWVUuVyloYj10EpdWvm78V2pdRXZd3GsuDA30ZdpdRSpdSmzL+Pfla0s7QppWYrpWKUUvkW81fG1MzrtFUp1dahN9Zal8kNk/DdDzQAfIAtQIs854wCPsh8fBvwTVm1z8muQ3cgIPPxyPJ4HRy9FpnnVQRWAOuA9la326LficbAJqBK5vPqVrfbouswExiZ+bgFcMjqdpfStbgaaAtsu8jxfsBCzBqoTsB6R963LHv4WSUatFmnrcQAAAJkSURBVNapgL1EQ04DgE8zH88DeqryV0i70OugtV6qtU7MfLoOs/ahPHLkdwJgIvAa4OC2aC7HketwPzBda30aQGsdU8ZtLAuOXAcNVMp8HMSFa4LKBa31CvJZy5TDAOAzbawDKiulahX2vmUZ8PMr0VD7YudordMBe4mG8sSR65DTvZj/ycujQq+FUioSqKO1/qUsG1bGHPmdaAI0UUqtVkqty6xgW944ch0mAEOVUtGYmYMPlU3TnE5R4whQtvXwS6xEg4tz+GdUSg0F2gPltWRogddCKeWBqb46vKwaZBFHfie8MMM63TCf+FYqpcK11mdKuW1lyZHrMAT4RGv9f0qpzpj1P+Faa1vpN8+pFCtWlmUPvyglGiioRIOLc+Q6oJS6BngWuEFrnVJGbStrhV2LikA4sEwpdQgzVvlTOUzcOvq38aPWOk1rfRDYjfkPoDxx5DrcC8wF0FqvBfwwNXbcjUNxJK+yDPhSosEo9DpkDmPMwAT78jhWa1fgtdBax2utQ7TWYVrrMEw+4watdbFriTgpR/425mOS+SilQjBDPAfKtJWlz5HrcAToCaCUao4J+O64ndZPwF2Zs3U6AfFa638Le1GZDeloKdEAOHwdXgcqAN9m5qyPaK1vsKzRpcTBa1HuOXgdFgG9lVI7gAzgCa11nHWtLnkOXofHgA+VUo9ghjCGl8NOIUqprzHDdyGZ+YrxgDeA1voDTP6iH7APSATuduh9y+G1EkIIkQ9ZaSuEEG5CAr4QQrgJCfhCCOEmJOALIYSbkIAvhBBuQgK+EEK4CQn4QgjhJiTgCyGEm/h/EuSptzspCHsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "# on extrait les probabilités d'appartenance\n",
    "proba_rf= modele_rf.predict_proba(x_test)[:,1]\n",
    "proba_knn=modele_knn.predict_proba(x_test)[:,1]\n",
    "\n",
    "# cas du modèle RF\n",
    "fpr, tpr, _ = roc_curve(y_test, proba_rf)\n",
    "plt.plot(fpr,tpr,\"b-.\", label=\"RF\")\n",
    "\n",
    "# cas du modèle kNN\n",
    "fpr, tpr, _ = roc_curve(y_test, proba_knn)\n",
    "plt.plot(fpr,tpr,\":\", label=\"kNN\")\n",
    "\n",
    "# modèle aléatoire\n",
    "plt.plot([0, 1], [0, 1],\"r-\", label=\"aléatoire\", )\n",
    "\n",
    "# modèle parfait\n",
    "plt.plot([0,0, 1], [0,1, 1], 'b--', label=\"parfait\")\n",
    "\n",
    "plt.xlim([-0.01, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### L’aire sous la courbe ROC\n",
    "La courbe ROC est un indicateur important mais on préfère souvent une valeur plutôt\n",
    "qu’une courbe afin de comparer nos modèles. Pour cela, on utilise l’aire sous la courbe\n",
    "ROC (AUC). Cette aire est calculée directement à partir de la courbe ROC. Ainsi, un\n",
    "modèle aléatoire aura une AUC de 0.5 et un modèle parfait aura une AUC de 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Aire sous la courbe ROC pour le modèle RF : 0.9148175035289374\n",
      "Aire sous la courbe ROC pour le modèle kNN : 0.6475418431135309\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "auc_modele_rf=roc_auc_score(y_test, modele_rf.predict_proba(x_test)[:,1])\n",
    "auc_modele_knn=roc_auc_score(y_test,modele_knn.predict_proba(x_test)[:,1])\n",
    "\n",
    "print(\"Aire sous la courbe ROC pour le modèle RF :\" ,auc_modele_rf)\n",
    "print(\"Aire sous la courbe ROC pour le modèle kNN :\" ,auc_modele_knn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### La validation croisée\n",
    "Jusqu’ici nous avons utilisé des indicateurs basés sur une seule occurrence de test. Ceci veut dire qu’on ne teste notre modèle que sur un seul échantillon.\n",
    "\n",
    "Une approche alternative souvent utilisée est la validation croisée. Celle-ci est en fait basée sur la répétition de l’estimation et de la validation sur des données différentes.\n",
    "\n",
    "Pour obtenir ce cv-score, on utilise :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC pour RF : 0.90 (+/- 0.07)\n",
      "AUC pour kNN : 0.61 (+/- 0.07)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "scores_rf = cross_val_score(modele_rf, x, y, cv=5, scoring='roc_auc')\n",
    "scores_knn = cross_val_score(modele_knn, x, y, cv=5, scoring='roc_auc')\n",
    "\n",
    "print(\"AUC pour RF : %.2f (+/- %.2f)\"% (scores_rf.mean(), scores_knn.std() * 2))\n",
    "print(\"AUC pour kNN : %.2f (+/- %.2f)\"% (scores_knn.mean(),scores_knn.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### La validation dans le cas continu\n",
    "\n",
    "Il s’agit du cas où on essaye de prédire une variable quantitative. Dans ce cas, on ne peut pas utiliser les indicateurs précédents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R2 : 0.9022665051489202\n",
      "MCE : 10798409.639312023\n",
      "RMCE : 3286.09337044948\n",
      "MAE : 934.490310534231\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.metrics import r2_score\n",
    "from sklearn.metrics import explained_variance_score\n",
    "from sklearn.metrics import median_absolute_error\n",
    "\n",
    "# le R², à la différence des autres scores, est généralmeent\n",
    "# calculé sur l’échantillon d’apprentissage\n",
    "print(\"R2 :\",r2_score(y_train_cont,modele_ridge.predict(x_train_cont)))\n",
    "\n",
    "# les autres indicateurs sont basés sur l’échantillon de validation\n",
    "print(\"MCE :\",mean_squared_error(y_test_cont,y_predict_cont))\n",
    "print(\"RMCE :\",np.sqrt(mean_squared_error(y_test_cont,y_predict_cont)))\n",
    "print(\"MAE :\",median_absolute_error(y_test_cont,y_predict_cont))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.4 L’ajustement des hyperparamètres d’un modèle\n",
    "\n",
    "L’une des tâches du data scientist est de trouver le meilleur modèle possible. La\n",
    "plupart des modèles de machine learning ont des hyperparamètres. Il s’agit de paramètres\n",
    "du modèle qui sont définis en amont de l’ajustement.\n",
    "\n",
    "Scikit-Learn propose une classe GridSearchCV permettant d’implémenter cette\n",
    "recherche d’hyperparamètres :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On va donc devoir définir les hyperparamètres que l’on souhaite tester. Pour cela,\n",
    "on utilisera un dictionnaire d’hyperparamètres, par exemple :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "dico_param= {\"max_depth\":[3,5,7,10], \"n_estimators\":[10,20,50,100]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On va encore utiliser l’accuracy pour valider notre modèle. Finalement, nous allons\n",
    "utiliser une validation croisée à cinq groupes pour valider les résultats.\n",
    "Le nouvel objet est le suivant :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "recherche_hyper = GridSearchCV(RandomForestClassifier(), \n",
    "                               dico_param, \n",
    "                               scoring=\"accuracy\",cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Une fois qu’on a créé cet objet, on peut lui joindre les données afin d’estimer les\n",
    "meilleurs paramètres du modèle.\n",
    "\n",
    "Cette étape peut être très longue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "       estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators='warn', n_jobs=None,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'max_depth': [3, 5, 7, 10], 'n_estimators': [10, 20, 50, 100]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recherche_hyper.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,\n",
       "       1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,\n",
       "       0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0,\n",
       "       0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,\n",
       "       0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0,\n",
       "       0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recherche_hyper.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Si nous revenons à notre exemple de données des télécommunications avec nos\n",
    "deux estimateurs par forêt aléatoire et par plus proches voisins. Nous allons faire\n",
    "varier les hyperparamètres de ces deux modèles pour trouver la meilleure combinaison\n",
    "en termes d’aire sous la courbe ROC :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# construction des dictionnaires d’hyperparamètres\n",
    "dico_param_rf={\"n_estimators\":[10,100,1000],\"max_depth\":[5,7,9]}\n",
    "dico_param_knn={\"n_neighbors\":[2,5,10,50],\"weights\":['uniform','distance']}\n",
    "\n",
    "modele_grid_rf=GridSearchCV(modele_rf,dico_param_rf,\n",
    "                            scoring=\"roc_auc\",cv=5)\n",
    "\n",
    "modele_grid_knn=GridSearchCV(modele_knn, dico_param_knn,\n",
    "                             scoring=\"roc_auc\",cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "       estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=None,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'n_estimators': [10, 100, 1000], 'max_depth': [5, 7, 9]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='roc_auc', verbose=0)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# estimation des paramètres et des meilleurs modèles\n",
    "modele_grid_rf.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "       estimator=KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=None, n_neighbors=5, p=2,\n",
       "           weights='uniform'),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'n_neighbors': [2, 5, 10, 50], 'weights': ['uniform', 'distance']},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='roc_auc', verbose=0)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# estimation des paramètres et des meilleurs modèles\n",
    "modele_grid_knn.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Meilleurs paramètres RF: {'max_depth': 7, 'n_estimators': 1000}\n",
      "AUC - RF: 0.9049216277517667\n",
      "Meilleurs paramètres kNN: {'n_neighbors': 10, 'weights': 'distance'}\n",
      "AUC - kNN: 0.5856173109003944\n"
     ]
    }
   ],
   "source": [
    "# affichage des AUC pour la combinaison obtenant\n",
    "# les meilleurs résultats\n",
    "print(\"Meilleurs paramètres RF:\", modele_grid_rf.best_params_)\n",
    "print(\"AUC - RF:\", modele_grid_rf.best_score_)\n",
    "print(\"Meilleurs paramètres kNN:\", modele_grid_knn.best_params_)\n",
    "print(\"AUC - kNN:\", modele_grid_knn.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On voit que malgré ce travail, les forêts aléatoires sont toujours plus efficaces que\n",
    "les plus proches voisins.\n",
    "Si le nombre d’hyperparamètres devient trop grand, on utilise alors une recherche\n",
    "aléatoire parmi les combinaisons avec la classe RandomSearchCV."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.5 La construction d’un pipeline de traitement\n",
    "\n",
    "Bien souvent vous allez être amené à enchaîner des traitements sur des données.\n",
    "On peut bien sûr développer son code de manière à suivre les étapes une à une mais il est souvent plus intéressant de créer des suites de traitements automatisées avec Scikit-Learn. \n",
    "\n",
    "Ces suites de traitements sont appelées pipeline. Ils simplifieront votre code et permettront de passer en production simplement.\n",
    "\n",
    "Ainsi, on va pouvoir faire une analyse en composantes principales suivies d’un\n",
    "algorithme de plus proches voisins directement dans un pipeline :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('acp', PCA(copy=True, iterated_power='auto', n_components=8, random_state=None,\n",
       "  svd_solver='auto', tol=0.0, whiten=False)), ('knn', KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=None, n_neighbors=5, p=2,\n",
       "           weights='uniform'))])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acp=PCA(n_components=8)\n",
    "knn=KNeighborsClassifier()\n",
    "pipe=Pipeline(steps=[(\"acp\",acp),(\"knn\",knn)])\n",
    "pipe.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.626537608388788"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_test,pipe.predict_proba(x_test)[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On a ainsi enchaîné deux traitements. Si on cherche des sorties liées à chacune\n",
    "des étapes, on pourra le faire simplement. Par exemple, si l’objectif est d’extraire la part de variances expliquées par les composantes de l’analyse en composantes principales, on fera :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9.87979773e-01, 3.21114661e-03, 2.73484112e-03, 2.69143298e-03,\n",
       "       1.68794539e-03, 4.27753608e-04, 4.21396719e-04, 3.96891395e-04])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.named_steps[\"acp\"].explained_variance_ratio_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Trouver la meilleure combinaison d’hyperparamètres dans un pipeline\n",
    "\n",
    "Essayons de trouver la meilleure combinaison d’hyperparamètres dans un pipeline.\n",
    "Dans le cadre de cet exemple, nous utiliserons les SVM (support vector machines,\n",
    "également appelés séparateurs à vaste marge ou machines à vecteurs de support).\n",
    "Ce sont des méthodes assez complexes dans leur principe mais simples dans leur\n",
    "mise en oeuvre.\n",
    "\n",
    "Les hyperparamètres d’un modèle de SVM sont assez nombreux. Les plus importants\n",
    "étant le noyau choisi (linéaire, polynomial, sigmoïd, RBF…), les paramètres de\n",
    "ces noyaux (le degré pour le cas polynomiale, gamma…) et le C pour la marge floue.\n",
    "\n",
    "Dans notre exemple, on utilisera la fonction make_pipeline qui est équivalente\n",
    "à la classe précédente avec quelques simplifications :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# construction du pipeline basé sur deux approches\n",
    "mon_pipe=make_pipeline(PCA(), SVC(probability=True))\n",
    "\n",
    "# construction du dictionnaire des paramètres\n",
    "# (attention utilisation de __)\n",
    "param_grid = dict(pca__n_components=[5, 10, x_train.shape[1]],\n",
    "                  svc__C= [1, 10, 100, 1000],\n",
    "                  svc__kernel= ['sigmoid', 'rbf'],\n",
    "                  svc__gamma= [0.001, 0.0001])\n",
    "\n",
    "# on construit l’objet GridSearch et on estime les hyper-paramètres\n",
    "# par validation croisée\n",
    "grid_search_mon_pipe = GridSearchCV(mon_pipe, param_grid = param_grid, scoring = \"roc_auc\", cv = 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=4, error_score='raise-deprecating',\n",
       "       estimator=Pipeline(memory=None,\n",
       "     steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=None, random_state=None,\n",
       "  svd_solver='auto', tol=0.0, whiten=False)), ('svc', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto_deprecated',\n",
       "  kernel='rbf', max_iter=-1, probability=True, random_state=None,\n",
       "  shrinking=True, tol=0.001, verbose=False))]),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'pca__n_components': [5, 10, 22], 'svc__C': [1, 10, 100, 1000], 'svc__kernel': ['sigmoid', 'rbf'], 'svc__gamma': [0.001, 0.0001]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='roc_auc', verbose=0)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search_mon_pipe.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'pca__n_components': 5,\n",
       " 'svc__C': 1,\n",
       " 'svc__gamma': 0.0001,\n",
       " 'svc__kernel': 'rbf'}"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# la meilleure combinaisons de paramètres est :\n",
    "grid_search_mon_pipe.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les meilleurs hyperparamètres obtenus en utilisant l’aire sous la courbe ROC sont\n",
    "la combinaison C = 10, gamma = 0.0001, un noyau RBF pour les SVM et dix composantes\n",
    "pour notre analyse en composantes principales.\n",
    "\n",
    "Dans ce code, on définit les hyperparamètres associés à une méthode du pipeline\n",
    "avec un double underscore : __.\n",
    "\n",
    "L’utilisation des pipelines de Scikit-Learn va devenir rapidement une étape cruciale\n",
    "de vos développements en Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.6 Passer en production votre modèle d’apprentissage supervisé\n",
    "\n",
    "#### Persistance de modèle avec Scikit-Learn\n",
    "\n",
    "Python possède plusieurs outils pour la persistance d’objets, c’est-à-dire pour stocker\n",
    "des objets dans des fichiers. Les objets de Scikit-Learn sont aussi dans cette\n",
    "situation. On utilise un format pickle qui aura l’extension .pkl.\n",
    "\n",
    "Par exemple, si nous voulons sauvegarder le dernier pipeline de traitement, nous\n",
    "allons utiliser :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['../data/modele_grid_pipe.pkl']"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.externals import joblib\n",
    "joblib.dump(grid_search_mon_pipe, '../data/modele_grid_pipe.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Une fois ce modèle stocké, on peut très bien le réutiliser dans un autre cadre. Si\n",
    "nous créons un nouveau notebook, nous allons utiliser :\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.externals import joblib\n",
    "grid_search_mon_pipe = joblib.load('../data/modele_grid_pipe.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut ensuite appliquer le modèle avec tous les paramètres qui ont été appris :\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "grid_search_mon_pipe.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "L’utilisation d’un fichier Pickle dans un notebook est une technique assez simple et courante."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}