symmetric_input_pair.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-Validation with Symmetric Pair-Input Data\n",
    "\n",
    "In the symmetric Pair-Input Data,  data instance shares common pair-members, which  result the strong dependencies among data samples and this violates the IID assumption. Thus, the modfied verison of cross-validaiton for symmetric Pair-Input Data is applied in this task.  The c-index is used as the evaluation metric.\n",
    "\n",
    "In such data, three possible types of pairs with respect to training set can be found:\n",
    "      \n",
    "      A – both members are present in training set\n",
    "      B – only one member is present in training set\n",
    "      C – neither member is present in training set\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                        =====  Implementation start  ======\n",
    "\n",
    "Note: All the three cross-validations are perfomed with normalized data and with out normalized. The z-score is used for normalization. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import necessary libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import operator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset\n",
    "def load_dataset(data):\n",
    "    df = pd.read_csv(data, header=None) # read dataset\n",
    "    return df\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimension of dataframe: (95, 41)\n",
      "\n",
      "data types: 0     int64\n",
      "1     int64\n",
      "2     int64\n",
      "3     int64\n",
      "4     int64\n",
      "5     int64\n",
      "6     int64\n",
      "7     int64\n",
      "8     int64\n",
      "9     int64\n",
      "10    int64\n",
      "11    int64\n",
      "12    int64\n",
      "13    int64\n",
      "14    int64\n",
      "15    int64\n",
      "16    int64\n",
      "17    int64\n",
      "18    int64\n",
      "19    int64\n",
      "20    int64\n",
      "21    int64\n",
      "22    int64\n",
      "23    int64\n",
      "24    int64\n",
      "25    int64\n",
      "26    int64\n",
      "27    int64\n",
      "28    int64\n",
      "29    int64\n",
      "30    int64\n",
      "31    int64\n",
      "32    int64\n",
      "33    int64\n",
      "34    int64\n",
      "35    int64\n",
      "36    int64\n",
      "37    int64\n",
      "38    int64\n",
      "39    int64\n",
      "40    int64\n",
      "dtype: object\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>94</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>51</td>\n",
       "      <td>54</td>\n",
       "      <td>54</td>\n",
       "      <td>74</td>\n",
       "      <td>21</td>\n",
       "      <td>43</td>\n",
       "      <td>61</td>\n",
       "      <td>...</td>\n",
       "      <td>52</td>\n",
       "      <td>26</td>\n",
       "      <td>55</td>\n",
       "      <td>47</td>\n",
       "      <td>225</td>\n",
       "      <td>140</td>\n",
       "      <td>43</td>\n",
       "      <td>30</td>\n",
       "      <td>132</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>12</td>\n",
       "      <td>5</td>\n",
       "      <td>19</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>20</td>\n",
       "      <td>...</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>7</td>\n",
       "      <td>32</td>\n",
       "      <td>23</td>\n",
       "      <td>11</td>\n",
       "      <td>7</td>\n",
       "      <td>26</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>88</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>59</td>\n",
       "      <td>58</td>\n",
       "      <td>55</td>\n",
       "      <td>76</td>\n",
       "      <td>24</td>\n",
       "      <td>53</td>\n",
       "      <td>57</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>32</td>\n",
       "      <td>56</td>\n",
       "      <td>52</td>\n",
       "      <td>227</td>\n",
       "      <td>146</td>\n",
       "      <td>45</td>\n",
       "      <td>35</td>\n",
       "      <td>135</td>\n",
       "      <td>105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>52</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>17</td>\n",
       "      <td>51</td>\n",
       "      <td>47</td>\n",
       "      <td>16</td>\n",
       "      <td>50</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>19</td>\n",
       "      <td>10</td>\n",
       "      <td>21</td>\n",
       "      <td>19</td>\n",
       "      <td>184</td>\n",
       "      <td>109</td>\n",
       "      <td>15</td>\n",
       "      <td>10</td>\n",
       "      <td>106</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>27</td>\n",
       "      <td>23</td>\n",
       "      <td>51</td>\n",
       "      <td>64</td>\n",
       "      <td>19</td>\n",
       "      <td>65</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>25</td>\n",
       "      <td>16</td>\n",
       "      <td>31</td>\n",
       "      <td>24</td>\n",
       "      <td>194</td>\n",
       "      <td>118</td>\n",
       "      <td>16</td>\n",
       "      <td>15</td>\n",
       "      <td>117</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0   1   2   3   4   5   6   7   8   9  ...   31  32  33  34   35   36  37  \\\n",
       "0  94   0  16  51  54  54  74  21  43  61 ...   52  26  55  47  225  140  43   \n",
       "1  21   0   6   8  12   5  19   5   6  20 ...    6   8  10   7   32   23  11   \n",
       "2  88   0  14  59  58  55  76  24  53  57 ...   57  32  56  52  227  146  45   \n",
       "3  52   0   5  18  17  51  47  16  50  15 ...   19  10  21  19  184  109  15   \n",
       "4  39   0  15  27  23  51  64  19  65  15 ...   25  16  31  24  194  118  16   \n",
       "\n",
       "   38   39   40  \n",
       "0  30  132   92  \n",
       "1   7   26   19  \n",
       "2  35  135  105  \n",
       "3  10  106   75  \n",
       "4  15  117  106  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#run load_dataset function\n",
    "df = load_dataset('Dataset/features.data')\n",
    "print('Dimension of dataframe:', df.shape)\n",
    "\n",
    "#check data types \n",
    "print('\\ndata types:',df.dtypes)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                             ==== Normalize data with z-score ====\n",
    "    \n",
    "Since every input feature is of 'int' data type, it should be change into 'float'  because when you try to convert them into normalized form it will  just 'int' type. So, the numbers like '1.23' and '1'  will be 1, though they are not the same. This will cause the difference in result then as it acutally should be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.800051</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.669472</td>\n",
       "      <td>1.937529</td>\n",
       "      <td>1.850945</td>\n",
       "      <td>0.820203</td>\n",
       "      <td>1.182565</td>\n",
       "      <td>0.591083</td>\n",
       "      <td>0.133767</td>\n",
       "      <td>2.191697</td>\n",
       "      <td>...</td>\n",
       "      <td>1.897792</td>\n",
       "      <td>1.666761</td>\n",
       "      <td>1.977547</td>\n",
       "      <td>1.811226</td>\n",
       "      <td>0.936060</td>\n",
       "      <td>1.042499</td>\n",
       "      <td>2.033321</td>\n",
       "      <td>1.904285</td>\n",
       "      <td>0.950928</td>\n",
       "      <td>0.863910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.146221</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.500156</td>\n",
       "      <td>-1.157820</td>\n",
       "      <td>-1.056642</td>\n",
       "      <td>-1.507873</td>\n",
       "      <td>-1.086588</td>\n",
       "      <td>-1.537936</td>\n",
       "      <td>-1.739502</td>\n",
       "      <td>-0.204895</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.451704</td>\n",
       "      <td>-1.039945</td>\n",
       "      <td>-1.459044</td>\n",
       "      <td>-1.520167</td>\n",
       "      <td>-1.677808</td>\n",
       "      <td>-1.608468</td>\n",
       "      <td>-0.836361</td>\n",
       "      <td>-0.930090</td>\n",
       "      <td>-1.673310</td>\n",
       "      <td>-1.676895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.557892</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.235546</td>\n",
       "      <td>2.513408</td>\n",
       "      <td>2.127858</td>\n",
       "      <td>0.867715</td>\n",
       "      <td>1.265080</td>\n",
       "      <td>0.990274</td>\n",
       "      <td>0.640056</td>\n",
       "      <td>1.957883</td>\n",
       "      <td>...</td>\n",
       "      <td>2.261868</td>\n",
       "      <td>2.568996</td>\n",
       "      <td>2.053916</td>\n",
       "      <td>2.227650</td>\n",
       "      <td>0.963147</td>\n",
       "      <td>1.178446</td>\n",
       "      <td>2.212676</td>\n",
       "      <td>2.520454</td>\n",
       "      <td>1.025199</td>\n",
       "      <td>1.316383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.104936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.717119</td>\n",
       "      <td>-0.437971</td>\n",
       "      <td>-0.710501</td>\n",
       "      <td>0.677668</td>\n",
       "      <td>0.068617</td>\n",
       "      <td>-0.074236</td>\n",
       "      <td>0.488169</td>\n",
       "      <td>-0.497162</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.505107</td>\n",
       "      <td>-0.739200</td>\n",
       "      <td>-0.618988</td>\n",
       "      <td>-0.520749</td>\n",
       "      <td>0.380782</td>\n",
       "      <td>0.340106</td>\n",
       "      <td>-0.477651</td>\n",
       "      <td>-0.560389</td>\n",
       "      <td>0.307247</td>\n",
       "      <td>0.272216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.419743</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.452509</td>\n",
       "      <td>0.209893</td>\n",
       "      <td>-0.295131</td>\n",
       "      <td>0.677668</td>\n",
       "      <td>0.769992</td>\n",
       "      <td>0.324955</td>\n",
       "      <td>1.247602</td>\n",
       "      <td>-0.497162</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.068216</td>\n",
       "      <td>0.163036</td>\n",
       "      <td>0.144699</td>\n",
       "      <td>-0.104325</td>\n",
       "      <td>0.516216</td>\n",
       "      <td>0.544027</td>\n",
       "      <td>-0.387973</td>\n",
       "      <td>0.055779</td>\n",
       "      <td>0.579574</td>\n",
       "      <td>1.351188</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         0    1         2         3         4         5         6         7   \\\n",
       "0  1.800051  0.0  1.669472  1.937529  1.850945  0.820203  1.182565  0.591083   \n",
       "1 -1.146221  0.0 -0.500156 -1.157820 -1.056642 -1.507873 -1.086588 -1.537936   \n",
       "2  1.557892  0.0  1.235546  2.513408  2.127858  0.867715  1.265080  0.990274   \n",
       "3  0.104936  0.0 -0.717119 -0.437971 -0.710501  0.677668  0.068617 -0.074236   \n",
       "4 -0.419743  0.0  1.452509  0.209893 -0.295131  0.677668  0.769992  0.324955   \n",
       "\n",
       "         8         9     ...           31        32        33        34  \\\n",
       "0  0.133767  2.191697    ...     1.897792  1.666761  1.977547  1.811226   \n",
       "1 -1.739502 -0.204895    ...    -1.451704 -1.039945 -1.459044 -1.520167   \n",
       "2  0.640056  1.957883    ...     2.261868  2.568996  2.053916  2.227650   \n",
       "3  0.488169 -0.497162    ...    -0.505107 -0.739200 -0.618988 -0.520749   \n",
       "4  1.247602 -0.497162    ...    -0.068216  0.163036  0.144699 -0.104325   \n",
       "\n",
       "         35        36        37        38        39        40  \n",
       "0  0.936060  1.042499  2.033321  1.904285  0.950928  0.863910  \n",
       "1 -1.677808 -1.608468 -0.836361 -0.930090 -1.673310 -1.676895  \n",
       "2  0.963147  1.178446  2.212676  2.520454  1.025199  1.316383  \n",
       "3  0.380782  0.340106 -0.477651 -0.560389  0.307247  0.272216  \n",
       "4  0.516216  0.544027 -0.387973  0.055779  0.579574  1.351188  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Normalize the data\n",
    "data_matrix = df.values# change to numpy array\n",
    "data_matrix = data_matrix.astype(float) # change to float type\n",
    "\n",
    "for i in range(len(data_matrix.T)): # transpose and calcualte 'mean' and 'stdv' row-wise\n",
    "    \n",
    "    # check if all elements in (ith) row are zeros \n",
    "    #  then  calculate 'mean' and 'stdv' for  other than ith rows\n",
    "    if np.all(data_matrix.T[i] == 0) == False:\n",
    "        mean = np.mean(data_matrix.T[i])\n",
    "        stdv = np.std(data_matrix.T[i])\n",
    "        \n",
    "        for j in range(len(data_matrix)) : # apply z-score in each element of df\n",
    "            data_matrix[j][i]= (data_matrix[j][i] - mean)/stdv\n",
    "        \n",
    "data_matrix = pd.DataFrame(data_matrix)  # change to pandas dataframe \n",
    "data_matrix.head()\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                    === target variable(label) is combined with input feature ====\n",
    " Since Knn algorithm is desinged in such a way that it takes  input features plus target variable(labels) together as the data matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.800051</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.669472</td>\n",
       "      <td>1.937529</td>\n",
       "      <td>1.850945</td>\n",
       "      <td>0.820203</td>\n",
       "      <td>1.182565</td>\n",
       "      <td>0.591083</td>\n",
       "      <td>0.133767</td>\n",
       "      <td>2.191697</td>\n",
       "      <td>...</td>\n",
       "      <td>1.666761</td>\n",
       "      <td>1.977547</td>\n",
       "      <td>1.811226</td>\n",
       "      <td>0.936060</td>\n",
       "      <td>1.042499</td>\n",
       "      <td>2.033321</td>\n",
       "      <td>1.904285</td>\n",
       "      <td>0.950928</td>\n",
       "      <td>0.863910</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.146221</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.500156</td>\n",
       "      <td>-1.157820</td>\n",
       "      <td>-1.056642</td>\n",
       "      <td>-1.507873</td>\n",
       "      <td>-1.086588</td>\n",
       "      <td>-1.537936</td>\n",
       "      <td>-1.739502</td>\n",
       "      <td>-0.204895</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.039945</td>\n",
       "      <td>-1.459044</td>\n",
       "      <td>-1.520167</td>\n",
       "      <td>-1.677808</td>\n",
       "      <td>-1.608468</td>\n",
       "      <td>-0.836361</td>\n",
       "      <td>-0.930090</td>\n",
       "      <td>-1.673310</td>\n",
       "      <td>-1.676895</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.557892</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.235546</td>\n",
       "      <td>2.513408</td>\n",
       "      <td>2.127858</td>\n",
       "      <td>0.867715</td>\n",
       "      <td>1.265080</td>\n",
       "      <td>0.990274</td>\n",
       "      <td>0.640056</td>\n",
       "      <td>1.957883</td>\n",
       "      <td>...</td>\n",
       "      <td>2.568996</td>\n",
       "      <td>2.053916</td>\n",
       "      <td>2.227650</td>\n",
       "      <td>0.963147</td>\n",
       "      <td>1.178446</td>\n",
       "      <td>2.212676</td>\n",
       "      <td>2.520454</td>\n",
       "      <td>1.025199</td>\n",
       "      <td>1.316383</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.104936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.717119</td>\n",
       "      <td>-0.437971</td>\n",
       "      <td>-0.710501</td>\n",
       "      <td>0.677668</td>\n",
       "      <td>0.068617</td>\n",
       "      <td>-0.074236</td>\n",
       "      <td>0.488169</td>\n",
       "      <td>-0.497162</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.739200</td>\n",
       "      <td>-0.618988</td>\n",
       "      <td>-0.520749</td>\n",
       "      <td>0.380782</td>\n",
       "      <td>0.340106</td>\n",
       "      <td>-0.477651</td>\n",
       "      <td>-0.560389</td>\n",
       "      <td>0.307247</td>\n",
       "      <td>0.272216</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.419743</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.452509</td>\n",
       "      <td>0.209893</td>\n",
       "      <td>-0.295131</td>\n",
       "      <td>0.677668</td>\n",
       "      <td>0.769992</td>\n",
       "      <td>0.324955</td>\n",
       "      <td>1.247602</td>\n",
       "      <td>-0.497162</td>\n",
       "      <td>...</td>\n",
       "      <td>0.163036</td>\n",
       "      <td>0.144699</td>\n",
       "      <td>-0.104325</td>\n",
       "      <td>0.516216</td>\n",
       "      <td>0.544027</td>\n",
       "      <td>-0.387973</td>\n",
       "      <td>0.055779</td>\n",
       "      <td>0.579574</td>\n",
       "      <td>1.351188</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          0    1         2         3         4         5         6         7  \\\n",
       "0  1.800051  0.0  1.669472  1.937529  1.850945  0.820203  1.182565  0.591083   \n",
       "1 -1.146221  0.0 -0.500156 -1.157820 -1.056642 -1.507873 -1.086588 -1.537936   \n",
       "2  1.557892  0.0  1.235546  2.513408  2.127858  0.867715  1.265080  0.990274   \n",
       "3  0.104936  0.0 -0.717119 -0.437971 -0.710501  0.677668  0.068617 -0.074236   \n",
       "4 -0.419743  0.0  1.452509  0.209893 -0.295131  0.677668  0.769992  0.324955   \n",
       "\n",
       "          8         9   ...          32        33        34        35  \\\n",
       "0  0.133767  2.191697   ...    1.666761  1.977547  1.811226  0.936060   \n",
       "1 -1.739502 -0.204895   ...   -1.039945 -1.459044 -1.520167 -1.677808   \n",
       "2  0.640056  1.957883   ...    2.568996  2.053916  2.227650  0.963147   \n",
       "3  0.488169 -0.497162   ...   -0.739200 -0.618988 -0.520749  0.380782   \n",
       "4  1.247602 -0.497162   ...    0.163036  0.144699 -0.104325  0.516216   \n",
       "\n",
       "         36        37        38        39        40  Target  \n",
       "0  1.042499  2.033321  1.904285  0.950928  0.863910       1  \n",
       "1 -1.608468 -0.836361 -0.930090 -1.673310 -1.676895       1  \n",
       "2  1.178446  2.212676  2.520454  1.025199  1.316383       0  \n",
       "3  0.340106 -0.477651 -0.560389  0.307247  0.272216       1  \n",
       "4  0.544027 -0.387973  0.055779  0.579574  1.351188       1  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# target variable(output)\n",
    "df_target = pd.read_csv('Dataset/labels.data', names=['Target'])\n",
    "data_matrix_target = pd.concat([data_matrix, df_target], axis=1)\n",
    "data_matrix_target.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>94</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>51</td>\n",
       "      <td>54</td>\n",
       "      <td>54</td>\n",
       "      <td>74</td>\n",
       "      <td>21</td>\n",
       "      <td>43</td>\n",
       "      <td>61</td>\n",
       "      <td>...</td>\n",
       "      <td>26</td>\n",
       "      <td>55</td>\n",
       "      <td>47</td>\n",
       "      <td>225</td>\n",
       "      <td>140</td>\n",
       "      <td>43</td>\n",
       "      <td>30</td>\n",
       "      <td>132</td>\n",
       "      <td>92</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>12</td>\n",
       "      <td>5</td>\n",
       "      <td>19</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>20</td>\n",
       "      <td>...</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>7</td>\n",
       "      <td>32</td>\n",
       "      <td>23</td>\n",
       "      <td>11</td>\n",
       "      <td>7</td>\n",
       "      <td>26</td>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>88</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>59</td>\n",
       "      <td>58</td>\n",
       "      <td>55</td>\n",
       "      <td>76</td>\n",
       "      <td>24</td>\n",
       "      <td>53</td>\n",
       "      <td>57</td>\n",
       "      <td>...</td>\n",
       "      <td>32</td>\n",
       "      <td>56</td>\n",
       "      <td>52</td>\n",
       "      <td>227</td>\n",
       "      <td>146</td>\n",
       "      <td>45</td>\n",
       "      <td>35</td>\n",
       "      <td>135</td>\n",
       "      <td>105</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>52</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>17</td>\n",
       "      <td>51</td>\n",
       "      <td>47</td>\n",
       "      <td>16</td>\n",
       "      <td>50</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>10</td>\n",
       "      <td>21</td>\n",
       "      <td>19</td>\n",
       "      <td>184</td>\n",
       "      <td>109</td>\n",
       "      <td>15</td>\n",
       "      <td>10</td>\n",
       "      <td>106</td>\n",
       "      <td>75</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>27</td>\n",
       "      <td>23</td>\n",
       "      <td>51</td>\n",
       "      <td>64</td>\n",
       "      <td>19</td>\n",
       "      <td>65</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>16</td>\n",
       "      <td>31</td>\n",
       "      <td>24</td>\n",
       "      <td>194</td>\n",
       "      <td>118</td>\n",
       "      <td>16</td>\n",
       "      <td>15</td>\n",
       "      <td>117</td>\n",
       "      <td>106</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    0  1   2   3   4   5   6   7   8   9   ...    32  33  34   35   36  37  \\\n",
       "0  94  0  16  51  54  54  74  21  43  61   ...    26  55  47  225  140  43   \n",
       "1  21  0   6   8  12   5  19   5   6  20   ...     8  10   7   32   23  11   \n",
       "2  88  0  14  59  58  55  76  24  53  57   ...    32  56  52  227  146  45   \n",
       "3  52  0   5  18  17  51  47  16  50  15   ...    10  21  19  184  109  15   \n",
       "4  39  0  15  27  23  51  64  19  65  15   ...    16  31  24  194  118  16   \n",
       "\n",
       "   38   39   40  Target  \n",
       "0  30  132   92       1  \n",
       "1   7   26   19       1  \n",
       "2  35  135  105       0  \n",
       "3  10  106   75       1  \n",
       "4  15  117  106       1  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " '''==== data with out normalization ====='''\n",
    "    \n",
    "# comibine input features with target class/label\n",
    "data_matrix_without_norm= pd.concat([df, df_target], axis=1)\n",
    "data_matrix_without_norm.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>P0</td>\n",
       "      <td>P9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>P6</td>\n",
       "      <td>P19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>P0</td>\n",
       "      <td>P4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>P5</td>\n",
       "      <td>P17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>P15</td>\n",
       "      <td>P17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1\n",
       "0   P0   P9\n",
       "1   P6  P19\n",
       "2   P0   P4\n",
       "3   P5  P17\n",
       "4  P15  P17"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read pairs-input in pandas dataframe\n",
    "input_pair = pd.read_csv('Dataset/pairs.data', header=None)\n",
    "input_pair.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                    ===  Knn from scratch implementation ====\n",
    "\n",
    "The three functions are used to perfome Knn classifier. The eculidan_dist() function calculates the eculidan distance between the two instances of data. The second function get_neighbours() finds the 'k' neighbours from the traiing dataset for the test instance, and the third function get_response() finds the 'class' for test instance  that belongs to higest votes with in the 'k' neigbours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#calculates eculidan distance between two data instances\n",
    "def eculidan_dist(data1, data2):\n",
    "    \n",
    "    distance = 0\n",
    "    length = len(data2) -1 # since last variable is target \n",
    "        \n",
    "    # loop through all  input variables (except target variables)\n",
    "    for x in range(length):\n",
    "        distance += np.power((data1[x]- data2[x]),2)\n",
    "    return np.sqrt(distance)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# finds k nearest  neighbors of  test instance \n",
    "def get_neighbours(trainingSet, testInstance, k):\n",
    "        distance = []\n",
    "        \n",
    "        # find distance  between  a test instance to all training instances\n",
    "        for x in range(len(trainingSet)):\n",
    "            dist = eculidan_dist(trainingSet[x], testInstance) # call eculidan_dist function\n",
    "            distance.append((trainingSet[x], dist)) \n",
    "        \n",
    "        # sort the distance in ascending order\n",
    "        distance.sort(key=operator.itemgetter(1))\n",
    "        \n",
    "\n",
    "        #store the  k neighbours\n",
    "        neighbours = []\n",
    "        for x in  range(k):\n",
    "            # takes only neighbours from distance list and append \n",
    "            neighbours.append(distance[x][0]) \n",
    "        return neighbours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# finds class that test instance belongs to. This is done by finding the  the most frequent class in the 'k' neigbours\n",
    "def get_response(neighbors):\n",
    "    classVotes = {} # for counting each class/label\n",
    "    \n",
    "    #counts each class in 'k' neighbours\n",
    "    for x in range(len(neighbors)):\n",
    "        response = neighbors[x][-1] # take last attribute(target variable) \n",
    "        if response in classVotes:\n",
    "            classVotes[response] +=1\n",
    "            \n",
    "        else:\n",
    "            classVotes[response] = 1\n",
    "    \n",
    "    # sorts classes with higest values\n",
    "    sortedVotes = sorted(classVotes.items(),key=operator.itemgetter(1), reverse=True) # appear as like [('b', 3), ('a', 2)]\n",
    "    return  sortedVotes[0][0] # take first neighbour(of higest votes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   ==== C-index Implementation start here ====="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#implementation  of C_index\n",
    "def c_index (true_labels, predictions):\n",
    "    n = 0\n",
    "    h_sum =0\n",
    "    \n",
    "    for i in range(len(true_labels)):\n",
    "        t = true_labels[i]\n",
    "        p = predictions[i]\n",
    "        \n",
    "        \n",
    "        for j in range(i+1, len(true_labels)):\n",
    "            nt = true_labels[j]\n",
    "            np = predictions[j]\n",
    "            \n",
    "            if( t!=nt):\n",
    "                n += 1\n",
    "                \n",
    "                if(p < np and t <  nt) or (p > np and t > nt):\n",
    "                    h_sum += 1\n",
    "                \n",
    "                elif(p == np):\n",
    "                    h_sum += 0.5\n",
    "                \n",
    "    c_index = h_sum/n\n",
    "    return c_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# testing c-index implementation\n",
    "true_labels = [-1, 1, 1, -1, 1]\n",
    "predictions = [0.60, 0.80, 0.75, 0.75, 0.70]\n",
    "c_index(true_labels, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                   ==== Leave one out cross_validation ====\n",
    "  This is the standard leave one out cross validaiton (with out modification). Each sample in the dataset becomes as test and remainig samples  become training set. So, the repeatation of this step (testing) goes to 'n' time. 'n' refers to total number of samples. This method belong to 'case-A' since the training set consists of both shared members of instance data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lev_one_out_cv(dataset, input_pair):\n",
    "    '''\n",
    "    dataset = dataset(input features + labels) in numpy array\n",
    "    input_pair = pair of protein  for input features in numpy array\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    \n",
    "    predicted_label = [] \n",
    "    true_label = []\n",
    "    \n",
    "    \n",
    "    # ====split data for test and train ===\n",
    "   \n",
    "    # use index for input-pair and dataset\n",
    "    for x in range(len(dataset)):\n",
    "        test_instance = dataset[x] # feature matrix for test instance of 'x' index\n",
    "        train_data = np.concatenate([dataset[0:x],dataset[1+x:]]) # train_data (other than test instance)\n",
    "        true_label.append(test_instance[-1]) # append class of test_instance\n",
    "          \n",
    "        \n",
    "       \n",
    "        \n",
    "    \n",
    "        '''=== Knn start from here ==='''\n",
    "        neighbours = get_neighbours(train_data, test_instance, 1) # call get_neighbours()\n",
    "        predict = get_response(neighbours) # call get_response()\n",
    "        predicted_label.append(predict)\n",
    "        \n",
    "    \n",
    "    ''' ==== compute c_index  ==== '''\n",
    "    c_index_val = c_index(true_label, predicted_label) # call c_index()\n",
    "        \n",
    "    return c_index_val\n",
    "            \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C_index with out data normalization: 0.762\n",
      "C_index with data normalization: 0.692\n"
     ]
    }
   ],
   "source": [
    "#run cross validataion function\n",
    "c_val= lev_one_out_cv(data_matrix_without_norm.values, input_pair.values)# with out normalization \n",
    "print('C_index with out data normalization: {0:.3f}'.format(c_val))\n",
    "\n",
    "c_val_1= lev_one_out_cv(data_matrix_target.values, input_pair.values) #normalized data\n",
    "print('C_index with data normalization: {0:.3f}'.format(c_val_1))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                     ==== Modified leave-one-out Cross validation ==== \n",
    "In this crossvalidation method, two for-loop is used. First loop is used to select 'test instance' and second loop is used for training set, which  removes the both members shared by 'test instance'. This cross-validation method belongs to the 'case-C' since in training set, there is no shared members at all.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_validation(dataset, input_pair):\n",
    "    '''\n",
    "    dataset = dataset(input features + labels) in numpy array\n",
    "    input_pair = pair of protein  for input features in numpy array\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    \n",
    "    predicted_class = [] # store predicted class\n",
    "    true_class = [] # store actual class of test instance\n",
    "    \n",
    "    \n",
    "    ''' split data for test and train '''\n",
    "    # iterates over data samples\n",
    "    for x in range(len(dataset)):# for test instance\n",
    "        '''x= index for numpy array'''\n",
    "        \n",
    "        test_instance = dataset[x] #test instance of 'x' index\n",
    "        train_data = [] # store train_data\n",
    "        true_class.append(test_instance[-1]) # append class/labels of test_instance\n",
    "    \n",
    "        \n",
    "        #compare 'input-pair' of test_instance with rest of data_points in dataset\n",
    "        #Remove those which are common one from training set\n",
    "        for y in range(len(dataset)):\n",
    "            if y!= x: # other then test instance index\n",
    "                if input_pair[y][0] != input_pair[x][0] and input_pair[y][0]!= input_pair[x][1] and input_pair[y][1]!= input_pair[x][0] and input_pair[y][1]!= input_pair[x][1]:\n",
    "                     train_data.append(dataset[y])\n",
    "                \n",
    "        \n",
    "        '''=== Knn start from here ==='''\n",
    "        train_dataset = np.array(train_data)\n",
    "        neighbours = get_neighbours(train_dataset, test_instance, 1) # call get_neighbours()\n",
    "        predict = get_response(neighbours) # call get_response()\n",
    "        predicted_class.append(predict)\n",
    "        \n",
    "    \n",
    "    \n",
    "    ''' ==== compute c_index  ==== '''\n",
    "    c_index_val = c_index(true_class, predicted_class) # call c_index()\n",
    "        \n",
    "    return c_index_val\n",
    "            \n",
    "            \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C_index with out data normalization: 0.631\n",
      "C_index with data normalization: 0.673\n"
     ]
    }
   ],
   "source": [
    "#run cross validataion function\n",
    "c_val= cross_validation(data_matrix_without_norm.values, input_pair.values) # with out normalized data\n",
    "print('C_index with out data normalization: {0:.3f}'.format(c_val))\n",
    "\n",
    "c_val_1= cross_validation(data_matrix_target.values, input_pair.values) #nomarlised data\n",
    "print('C_index with data normalization: {0:.3f}'.format(c_val_1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observation:\n",
    "1.Normal leave_one_out crossvalidation produces greater c_index than modified cross_validation. The reason behind this is in 'normal leave_one_out' cv  there is information about test instance in training set. Thus, leakage of information  about test instance is happend in normal leave_one_out. On the other hand, in modified cv there is no information leakege of test instance since we have removed all the shared members of test instance in training set.\n",
    "\n",
    "2.Unmodified cross_validation is suitable for case_A where all the two shared members of test instance are found in training set, while modified cross_validaiton is suitable for case_C since there is no shared member of test instance in trainig set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                                ==== Cross_validation for case_B =====\n",
    "In this task, the cross_validation suitable for the 'case_B'(=only one member is in training set) is designed.\n",
    "The previous cross_validation methods are not appropriate for the 'case_B' since unmodified crossvalidation method has 2 shared members of test instance in training set. Likewise, in the previous modified cross-validation there is no shared members in the trainig set.  \n",
    "\n",
    "In this cross-validation, the training set should  have only one(out of two) member of test instance. This can be done by predicting the same 'test instance' two time ( first-time:  trianing dataset contains only first member and second-time: the training dataset contains only second member or vice versa )\n",
    "\n",
    "\n",
    "For example, if S(a,b), 'S' is the test instance belongs to (a,b) pairs then, this same test should be tested/predicted in two times with two difference training dataset, which is described as below:\n",
    "   \n",
    "      In the first prediction, the train data contains first member i.e. a (lets say)\n",
    "      In second prediction,the train data contains second member i.e. b (lets say)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_validation_b(dataset, input_pair):\n",
    "    '''\n",
    "    dataset = dataset(input features + labels) in numpy array\n",
    "    input_pair = pair of protein  for input features in numpy array\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    \n",
    "    predicted_class = [] \n",
    "    true_class = []\n",
    "    \n",
    "    \n",
    "    # ====split data for test and train ===\n",
    "   \n",
    "    # use index for input-pair and dataset\n",
    "    for x in range(len(dataset)):# for test instance \n",
    "        test_instance = dataset[x] #test instance of 'x' index\n",
    "        train_data_1 = [] #train_data that includes first member of test instance\n",
    "        train_data_2 = [] # store train_data hat includes second member of test instance\n",
    "        true_class.append(test_instance[-1]) # append class of test_instance\n",
    "        true_class.append(test_instance[-1]) # append class of test_instance\n",
    "    \n",
    "        #print(input_pair[x][0], input_pair[x][1])\n",
    "        \n",
    "        #compare 'input-pair' of test_instance with rest of data_points in dataset\n",
    "        #train data should have one  member\n",
    "        \n",
    "        for y in range(len(dataset)):\n",
    "            if y!= x:  # other than test instance\n",
    "                \n",
    "                # compeltly no shared members\n",
    "                if input_pair[y][0] != input_pair[x][0] and input_pair[y][0]!= input_pair[x][1] and input_pair[y][1]!= input_pair[x][0] and input_pair[y][1]!= input_pair[x][1]:\n",
    "                     train_data_1.append(dataset[y])\n",
    "                    #train_data.append((input_pair[y][0],input_pair[y][1]))\n",
    "                        \n",
    "                 \n",
    "                # take only one member (first member) in training dataset\n",
    "                elif input_pair[y][0] == input_pair[x][0] or input_pair[y][1]== input_pair[x][0]: \n",
    "                    \n",
    "                    train_data_1.append(dataset[y])\n",
    "                    \n",
    "                \n",
    "        \n",
    "        \n",
    "        '''=== Knn start from here ==='''\n",
    "        train_dataset = np.array(train_data_1)\n",
    "        neighbours = get_neighbours(train_dataset, test_instance, 1) # call get_neighbours()\n",
    "        predict = get_response(neighbours) # call get_response()\n",
    "        predicted_class.append(predict)\n",
    "        \n",
    "        for y in range(len(dataset)):\n",
    "            if y!= x:  # other than test instance\n",
    "                \n",
    "                # compeltly no shared members\n",
    "                if input_pair[y][0] != input_pair[x][0] and input_pair[y][0]!= input_pair[x][1] and input_pair[y][1]!= input_pair[x][0] and input_pair[y][1]!= input_pair[x][1]:\n",
    "                     train_data_2.append(dataset[y])\n",
    "                    #train_data.append((input_pair[y][0],input_pair[y][1]))\n",
    "                        \n",
    "                    \n",
    "                # take only one member(second member) in training dataset\n",
    "                elif input_pair[y][0] == input_pair[x][1] or input_pair[y][1]== input_pair[x][1]: \n",
    "                    \n",
    "                    train_data_2.append(dataset[y])\n",
    "         \n",
    "  \n",
    "        '''=== Knn start from here ==='''\n",
    "        train_dataset = np.array(train_data_2)\n",
    "        neighbours = get_neighbours(train_dataset, test_instance, 1) # call get_neighbours()\n",
    "        predict = get_response(neighbours) # call get_response()\n",
    "        predicted_class.append(predict)\n",
    "        \n",
    "    \n",
    "    \n",
    "    ''' ==== compute c_index  ==== '''\n",
    "    c_index_val = c_index(true_class, predicted_class) # call c_index()\n",
    "        \n",
    "    return c_index_val\n",
    "            \n",
    "            \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C_index with out data normalization: 0.697\n",
      "C_index with data normalization: 0.690\n"
     ]
    }
   ],
   "source": [
    "#run cross validataion function\n",
    "c_val= cross_validation_b(data_matrix_without_norm.values, input_pair.values) # with out normalized data\n",
    "print('C_index with out data normalization: {0:.3f}'.format(c_val))\n",
    "\n",
    "c_val_1= cross_validation_b(data_matrix_target.values, input_pair.values) #nomarlised data\n",
    "print('C_index with data normalization: {0:.3f}'.format(c_val_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observation\n",
    "The c_index(0.697) obtained from third cross-validation(case-B) is beteween the unmodifed cross-validation(case-A) c-index (0.762) and  first modified cross-validation(case-C) c-index(0.631). The trainig set from this cross_validataion  still consists some information (one member) of test instance. Thus, c-index is greter than the previous modified cv version, which consits no infroamtion about test instance.  And, this c_index is less than the c-index from unmodified cross-validation because the infomation about test instance in training set from unmodified cross-validation has more( consist two members of test instance).\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}