AlphaWave Data Machine Learning to Predict Earnings for Stocks using Support-vector Machines example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning to Predict Earnings for Stocks: Support-vector Machines\n",
    "\n",
    "**Hugh Donnelly, CFA**<br> \n",
    "*AlphaWave Data*\n",
    "\n",
    "**September 2021**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "In this article, we are going to cover the support-vector machine (SVM) which is an incredibly powerful algorithm that can be used in both classification and regression settings. Let's begin by laying down the theoretical foundation of the algorithm.\n",
    "\n",
    "Jupyter Notebooks are available on [Google Colab](https://colab.research.google.com/drive/1bQCUij8nZwUAkSODja53R1sNWABauBl6?usp=sharing) and [Github](https://github.com/AlphaWaveData/Jupyter-Notebooks/blob/master/AlphaWave%20Data%20Machine%20Learning%20to%20Predict%20Earnings%20for%20Stocks%20using%20Support-vector%20Machines%20example.ipynb).\n",
    "\n",
    "For this project, we use several Python-based scientific computing technologies listed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Machine Learning Overview</h4>  \n",
    "<img src='ML Photos/1_SVM_ML_Graph.PNG'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So where does SVM fit among other [machine learning algorithms](https://hdonnelly6.medium.com/list/machine-learning-for-investing-7f2690bb1826)?\n",
    "\n",
    "SVM is in the supervised learning family. Supervised learning is applied when we want to map new input data to some output data.  In the context of classification, it will assign a label to some input data called X (i.e. independent features).  In a [regression](https://hdonnelly6.medium.com/introduction-to-machine-learning-regression-fee4200132f0), we map the input data X to some continuous output variable Y like in a single variant function, y = mx + b. The goal in supervised learning is to approximate a function that would take new input data X and predict output variable Y for that data. We train our algorithm in supervised learning, meaning we supervise it on the training data, and then we fine tune the algorithm so it performs well on the test data we feed it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build up the motivation and intuition for SVM.  Assume we have two classes defined by blue circles and red squares on a two dimensional plane.  How can we split this data so that the two classes are well separated?  We can start drawing lines, but you can see there are many options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>SVM used in Calssification and Regression</h4>  \n",
    "<img src='ML Photos/2_SVM_Categorize_Graph.PNG'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SVMs define the best line as the line that maximizes the margin between two classes.  This line is the optimal hyperplane as seen in the below graph.  However, we will likely be dealing with more than two dimensions in the SVM and the word hyperplane will be more accurate in this setting.  Keep in mind that we need to select the hyperplane that not only classifies this data, but also generalizes well for the new data points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Optimal Hyperplane</h4>  \n",
    "<img src='ML Photos/3_SVM_Hyperplane_Graph.PNG'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look closely, the optimal hyperplane is identified by the maximum margin that is between boundary lines of the classes.  It is exactly on those class boundary lines where support-vectors exist.  So, support-vectors are just points that lie on the two margins.  Notice that support-vectors solely determine the maximum margin and the optimal hyperplane.  The points that are outside the support-vectors do not influence the position of that hyperplane.\n",
    "\n",
    "As we can see, the main objective in SVM is to find the optimal hyperplane that correctly classifies data points between classes.  Dimension of this hyperplane is equal to the number of features minus one.  In our example where we had two features, the hyperplane would be a one dimensional line.  For three features, the hyperplane would be a two dimensional line.\n",
    "\n",
    "In our example, two classes were linearly separable.  What if the data looks like the below graph where one class is encircled by another class?  Clearly no matter how hard we try no line will be able to separate the data into two classes.  So this is where the kernel trick comes into play."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Using Kernal to convert linear data into higher dimension space</h4>  \n",
    "<img src='ML Photos/4_SVM_Hyperplane_Circle_Data_Graph.PNG'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The kernel transforms our input space into higher dimensional space where we can separate the classes.  Put another way, the kernel takes low dimensional space and transforms it into higher dimensional space where the data can be linearly separated as can be seen in the graph.\n",
    "\n",
    "After the data is transformed into the three dimensions, we can easily separate the data using a two dimensional hyperplane.  In practice, we often work with higher dimensions so this technique scales as well.  \n",
    "\n",
    "For those interested in kernel's mathematical properties, the kernel function acts as a modified dot product which takes input vectors in the original space and returns dot product of the transformed vectors in the higher dimensional space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initial Setup\n",
    "Now let's retrieve simulated quarterly fundamentals data for anonymized members of the S&P 500 from a saved pickle file we will use in this analysis.  This pickle file contains 51 features that we will use to predict the direction of the next quarter's earnings based on the current quarter's fundamental data.\n",
    "\n",
    "If you wish, you can also use real financial data provided by [AlphaWave Data](https://www.alphawavedata.com/) in this analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPS</th>\n",
       "      <th>change in EPS</th>\n",
       "      <th>Account Receivable Turnover</th>\n",
       "      <th>Current Ratio</th>\n",
       "      <th>Quick Ratio</th>\n",
       "      <th>Inventory Turnover</th>\n",
       "      <th>Total Debt To Equity</th>\n",
       "      <th>ROA</th>\n",
       "      <th>ROE</th>\n",
       "      <th>Gross Profit Margin</th>\n",
       "      <th>...</th>\n",
       "      <th>Change in Equity to Fixed Assets</th>\n",
       "      <th>Change in Sales to Total Assets</th>\n",
       "      <th>Change in EBIT to revenue</th>\n",
       "      <th>Change in Profit margin</th>\n",
       "      <th>Change in Sales to Inventory</th>\n",
       "      <th>Change in Sales to Working capital</th>\n",
       "      <th>Change in R&amp;D to Revenue</th>\n",
       "      <th>Change in working cap to Assets</th>\n",
       "      <th>Change in Operating Income or Losses</th>\n",
       "      <th>Change in EBITDA Margin</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.044</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.798445</td>\n",
       "      <td>3.150561</td>\n",
       "      <td>2.064582</td>\n",
       "      <td>22.240825</td>\n",
       "      <td>1325.517988</td>\n",
       "      <td>8.883952</td>\n",
       "      <td>33.958244</td>\n",
       "      <td>25.273754</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.424</td>\n",
       "      <td>0.380000</td>\n",
       "      <td>24.727972</td>\n",
       "      <td>2.700914</td>\n",
       "      <td>1.842956</td>\n",
       "      <td>21.072092</td>\n",
       "      <td>1347.030183</td>\n",
       "      <td>12.324957</td>\n",
       "      <td>44.042406</td>\n",
       "      <td>27.514299</td>\n",
       "      <td>...</td>\n",
       "      <td>-23.494239</td>\n",
       "      <td>-6.434533</td>\n",
       "      <td>29.515626</td>\n",
       "      <td>19.434621</td>\n",
       "      <td>-0.686169</td>\n",
       "      <td>52.633164</td>\n",
       "      <td>1.116427</td>\n",
       "      <td>-3.893287</td>\n",
       "      <td>21.181899</td>\n",
       "      <td>28.212400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.934</td>\n",
       "      <td>0.510000</td>\n",
       "      <td>24.581375</td>\n",
       "      <td>2.704947</td>\n",
       "      <td>1.884968</td>\n",
       "      <td>21.077957</td>\n",
       "      <td>1345.780617</td>\n",
       "      <td>14.640880</td>\n",
       "      <td>48.929598</td>\n",
       "      <td>28.984562</td>\n",
       "      <td>...</td>\n",
       "      <td>1.892533</td>\n",
       "      <td>-1.163430</td>\n",
       "      <td>12.278989</td>\n",
       "      <td>19.276009</td>\n",
       "      <td>-1.082365</td>\n",
       "      <td>-4.937915</td>\n",
       "      <td>-6.448939</td>\n",
       "      <td>0.917992</td>\n",
       "      <td>10.972702</td>\n",
       "      <td>2.175614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.084</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>24.166449</td>\n",
       "      <td>2.537465</td>\n",
       "      <td>1.756469</td>\n",
       "      <td>20.245552</td>\n",
       "      <td>1406.426471</td>\n",
       "      <td>16.180438</td>\n",
       "      <td>58.727070</td>\n",
       "      <td>28.999023</td>\n",
       "      <td>...</td>\n",
       "      <td>-27.764985</td>\n",
       "      <td>-0.219476</td>\n",
       "      <td>-0.933605</td>\n",
       "      <td>-0.300019</td>\n",
       "      <td>1.578928</td>\n",
       "      <td>52.635740</td>\n",
       "      <td>-0.104461</td>\n",
       "      <td>-7.192905</td>\n",
       "      <td>-1.151032</td>\n",
       "      <td>6.219108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.064</td>\n",
       "      <td>-1.020000</td>\n",
       "      <td>22.217841</td>\n",
       "      <td>3.063550</td>\n",
       "      <td>2.010260</td>\n",
       "      <td>17.716250</td>\n",
       "      <td>1402.460965</td>\n",
       "      <td>16.017779</td>\n",
       "      <td>59.725389</td>\n",
       "      <td>26.217002</td>\n",
       "      <td>...</td>\n",
       "      <td>17.893434</td>\n",
       "      <td>-7.214681</td>\n",
       "      <td>-24.290138</td>\n",
       "      <td>-32.450923</td>\n",
       "      <td>-9.127981</td>\n",
       "      <td>-51.242550</td>\n",
       "      <td>23.044946</td>\n",
       "      <td>-1.773955</td>\n",
       "      <td>-29.752363</td>\n",
       "      <td>-14.951802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.914</td>\n",
       "      <td>0.306168</td>\n",
       "      <td>14.783556</td>\n",
       "      <td>2.844682</td>\n",
       "      <td>2.526065</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1399.524214</td>\n",
       "      <td>4.393086</td>\n",
       "      <td>19.923301</td>\n",
       "      <td>76.257884</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.423973</td>\n",
       "      <td>6.126118</td>\n",
       "      <td>69.289285</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-19.917751</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-8.792037</td>\n",
       "      <td>10.821108</td>\n",
       "      <td>6.053170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.784</td>\n",
       "      <td>-0.130000</td>\n",
       "      <td>14.864186</td>\n",
       "      <td>2.843043</td>\n",
       "      <td>2.631217</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1401.624076</td>\n",
       "      <td>4.715484</td>\n",
       "      <td>20.690754</td>\n",
       "      <td>78.124570</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-5.681214</td>\n",
       "      <td>18.362582</td>\n",
       "      <td>-15.160033</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.456669</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.223852</td>\n",
       "      <td>11.638151</td>\n",
       "      <td>10.519497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.634</td>\n",
       "      <td>-0.150000</td>\n",
       "      <td>15.180999</td>\n",
       "      <td>3.065134</td>\n",
       "      <td>2.847354</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1388.890037</td>\n",
       "      <td>-0.439290</td>\n",
       "      <td>0.712332</td>\n",
       "      <td>80.462786</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-4.431060</td>\n",
       "      <td>-2.815343</td>\n",
       "      <td>-24.099620</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-22.544284</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.832590</td>\n",
       "      <td>-7.121654</td>\n",
       "      <td>0.428377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.674</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>16.602697</td>\n",
       "      <td>3.221663</td>\n",
       "      <td>2.807657</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1385.848914</td>\n",
       "      <td>4.210020</td>\n",
       "      <td>19.787080</td>\n",
       "      <td>73.175611</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.264846</td>\n",
       "      <td>-26.519779</td>\n",
       "      <td>9.836592</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-20.725509</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-4.097929</td>\n",
       "      <td>-26.714389</td>\n",
       "      <td>-17.679338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.784</td>\n",
       "      <td>0.110000</td>\n",
       "      <td>16.641741</td>\n",
       "      <td>3.222968</td>\n",
       "      <td>2.735622</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1389.848624</td>\n",
       "      <td>7.314623</td>\n",
       "      <td>27.510720</td>\n",
       "      <td>73.220111</td>\n",
       "      <td>...</td>\n",
       "      <td>-11.098760</td>\n",
       "      <td>10.544785</td>\n",
       "      <td>-18.447261</td>\n",
       "      <td>-19.839339</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.637515</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.854259</td>\n",
       "      <td>-9.847699</td>\n",
       "      <td>-13.907141</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4545 rows × 51 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       EPS  change in EPS  Account Receivable Turnover  Current Ratio  \\\n",
       "ID                                                                      \n",
       "0    3.044            NaN                    26.798445       3.150561   \n",
       "0    3.424       0.380000                    24.727972       2.700914   \n",
       "0    3.934       0.510000                    24.581375       2.704947   \n",
       "0    4.084       0.150000                    24.166449       2.537465   \n",
       "0    3.064      -1.020000                    22.217841       3.063550   \n",
       "..     ...            ...                          ...            ...   \n",
       "504  1.914       0.306168                    14.783556       2.844682   \n",
       "504  1.784      -0.130000                    14.864186       2.843043   \n",
       "504  1.634      -0.150000                    15.180999       3.065134   \n",
       "504  1.674       0.040000                    16.602697       3.221663   \n",
       "504  1.784       0.110000                    16.641741       3.222968   \n",
       "\n",
       "     Quick Ratio  Inventory Turnover  Total Debt To Equity        ROA  \\\n",
       "ID                                                                      \n",
       "0       2.064582           22.240825           1325.517988   8.883952   \n",
       "0       1.842956           21.072092           1347.030183  12.324957   \n",
       "0       1.884968           21.077957           1345.780617  14.640880   \n",
       "0       1.756469           20.245552           1406.426471  16.180438   \n",
       "0       2.010260           17.716250           1402.460965  16.017779   \n",
       "..           ...                 ...                   ...        ...   \n",
       "504     2.526065                 NaN           1399.524214   4.393086   \n",
       "504     2.631217                 NaN           1401.624076   4.715484   \n",
       "504     2.847354                 NaN           1388.890037  -0.439290   \n",
       "504     2.807657                 NaN           1385.848914   4.210020   \n",
       "504     2.735622                 NaN           1389.848624   7.314623   \n",
       "\n",
       "           ROE  Gross Profit Margin  ...  Change in Equity to Fixed Assets  \\\n",
       "ID                                   ...                                     \n",
       "0    33.958244            25.273754  ...                               NaN   \n",
       "0    44.042406            27.514299  ...                        -23.494239   \n",
       "0    48.929598            28.984562  ...                          1.892533   \n",
       "0    58.727070            28.999023  ...                        -27.764985   \n",
       "0    59.725389            26.217002  ...                         17.893434   \n",
       "..         ...                  ...  ...                               ...   \n",
       "504  19.923301            76.257884  ...                               NaN   \n",
       "504  20.690754            78.124570  ...                               NaN   \n",
       "504   0.712332            80.462786  ...                               NaN   \n",
       "504  19.787080            73.175611  ...                               NaN   \n",
       "504  27.510720            73.220111  ...                        -11.098760   \n",
       "\n",
       "     Change in Sales to Total Assets  Change in EBIT to revenue  \\\n",
       "ID                                                                \n",
       "0                                NaN                        NaN   \n",
       "0                          -6.434533                  29.515626   \n",
       "0                          -1.163430                  12.278989   \n",
       "0                          -0.219476                  -0.933605   \n",
       "0                          -7.214681                 -24.290138   \n",
       "..                               ...                        ...   \n",
       "504                         4.423973                   6.126118   \n",
       "504                        -5.681214                  18.362582   \n",
       "504                        -4.431060                  -2.815343   \n",
       "504                        -0.264846                 -26.519779   \n",
       "504                        10.544785                 -18.447261   \n",
       "\n",
       "     Change in Profit margin  Change in Sales to Inventory  \\\n",
       "ID                                                           \n",
       "0                        NaN                           NaN   \n",
       "0                  19.434621                     -0.686169   \n",
       "0                  19.276009                     -1.082365   \n",
       "0                  -0.300019                      1.578928   \n",
       "0                 -32.450923                     -9.127981   \n",
       "..                       ...                           ...   \n",
       "504                69.289285                           NaN   \n",
       "504               -15.160033                           NaN   \n",
       "504               -24.099620                           NaN   \n",
       "504                 9.836592                           NaN   \n",
       "504               -19.839339                           NaN   \n",
       "\n",
       "     Change in Sales to Working capital  Change in R&D to Revenue  \\\n",
       "ID                                                                  \n",
       "0                                   NaN                       NaN   \n",
       "0                             52.633164                  1.116427   \n",
       "0                             -4.937915                 -6.448939   \n",
       "0                             52.635740                 -0.104461   \n",
       "0                            -51.242550                 23.044946   \n",
       "..                                  ...                       ...   \n",
       "504                          -19.917751                       NaN   \n",
       "504                            5.456669                       NaN   \n",
       "504                          -22.544284                       NaN   \n",
       "504                          -20.725509                       NaN   \n",
       "504                            0.637515                       NaN   \n",
       "\n",
       "     Change in working cap to Assets  Change in Operating Income or Losses  \\\n",
       "ID                                                                           \n",
       "0                                NaN                                   NaN   \n",
       "0                          -3.893287                             21.181899   \n",
       "0                           0.917992                             10.972702   \n",
       "0                          -7.192905                             -1.151032   \n",
       "0                          -1.773955                            -29.752363   \n",
       "..                               ...                                   ...   \n",
       "504                        -8.792037                             10.821108   \n",
       "504                        -0.223852                             11.638151   \n",
       "504                        -0.832590                             -7.121654   \n",
       "504                        -4.097929                            -26.714389   \n",
       "504                         2.854259                             -9.847699   \n",
       "\n",
       "     Change in EBITDA Margin  \n",
       "ID                            \n",
       "0                        NaN  \n",
       "0                  28.212400  \n",
       "0                   2.175614  \n",
       "0                   6.219108  \n",
       "0                 -14.951802  \n",
       "..                       ...  \n",
       "504                 6.053170  \n",
       "504                10.519497  \n",
       "504                 0.428377  \n",
       "504               -17.679338  \n",
       "504               -13.907141  \n",
       "\n",
       "[4545 rows x 51 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load equity dataframe from the saved pickle file\n",
    "data = pd.read_pickle(\"./svm_data.pkl\")\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin by outlining the steps we will take to make this prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Earnings movement prediction\n",
    "\n",
    "#### Forecast direction of next quarter earnings based on accounting information of the current quarter\n",
    "\n",
    "#### Steps:\n",
    "- Enhance data with additional information\n",
    "- Preprocess the data\n",
    "- Apply Support-vector Machines on our dataset\n",
    "- Try to improve our results through [PCA](https://hdonnelly6.medium.com/machine-learning-for-esg-stock-trading-pca-and-clustering-ebe6077fc8f0)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPS</th>\n",
       "      <th>change in EPS</th>\n",
       "      <th>Account Receivable Turnover</th>\n",
       "      <th>Current Ratio</th>\n",
       "      <th>Quick Ratio</th>\n",
       "      <th>Inventory Turnover</th>\n",
       "      <th>Total Debt To Equity</th>\n",
       "      <th>ROA</th>\n",
       "      <th>ROE</th>\n",
       "      <th>Gross Profit Margin</th>\n",
       "      <th>...</th>\n",
       "      <th>Change in Equity to Fixed Assets</th>\n",
       "      <th>Change in Sales to Total Assets</th>\n",
       "      <th>Change in EBIT to revenue</th>\n",
       "      <th>Change in Profit margin</th>\n",
       "      <th>Change in Sales to Inventory</th>\n",
       "      <th>Change in Sales to Working capital</th>\n",
       "      <th>Change in R&amp;D to Revenue</th>\n",
       "      <th>Change in working cap to Assets</th>\n",
       "      <th>Change in Operating Income or Losses</th>\n",
       "      <th>Change in EBITDA Margin</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.044</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.798445</td>\n",
       "      <td>3.150561</td>\n",
       "      <td>2.064582</td>\n",
       "      <td>22.240825</td>\n",
       "      <td>1325.517988</td>\n",
       "      <td>8.883952</td>\n",
       "      <td>33.958244</td>\n",
       "      <td>25.273754</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.424</td>\n",
       "      <td>0.38</td>\n",
       "      <td>24.727972</td>\n",
       "      <td>2.700914</td>\n",
       "      <td>1.842956</td>\n",
       "      <td>21.072092</td>\n",
       "      <td>1347.030183</td>\n",
       "      <td>12.324957</td>\n",
       "      <td>44.042406</td>\n",
       "      <td>27.514299</td>\n",
       "      <td>...</td>\n",
       "      <td>-23.494239</td>\n",
       "      <td>-6.434533</td>\n",
       "      <td>29.515626</td>\n",
       "      <td>19.434621</td>\n",
       "      <td>-0.686169</td>\n",
       "      <td>52.633164</td>\n",
       "      <td>1.116427</td>\n",
       "      <td>-3.893287</td>\n",
       "      <td>21.181899</td>\n",
       "      <td>28.212400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.934</td>\n",
       "      <td>0.51</td>\n",
       "      <td>24.581375</td>\n",
       "      <td>2.704947</td>\n",
       "      <td>1.884968</td>\n",
       "      <td>21.077957</td>\n",
       "      <td>1345.780617</td>\n",
       "      <td>14.640880</td>\n",
       "      <td>48.929598</td>\n",
       "      <td>28.984562</td>\n",
       "      <td>...</td>\n",
       "      <td>1.892533</td>\n",
       "      <td>-1.163430</td>\n",
       "      <td>12.278989</td>\n",
       "      <td>19.276009</td>\n",
       "      <td>-1.082365</td>\n",
       "      <td>-4.937915</td>\n",
       "      <td>-6.448939</td>\n",
       "      <td>0.917992</td>\n",
       "      <td>10.972702</td>\n",
       "      <td>2.175614</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 51 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      EPS  change in EPS  Account Receivable Turnover  Current Ratio  \\\n",
       "ID                                                                     \n",
       "0   3.044            NaN                    26.798445       3.150561   \n",
       "0   3.424           0.38                    24.727972       2.700914   \n",
       "0   3.934           0.51                    24.581375       2.704947   \n",
       "\n",
       "    Quick Ratio  Inventory Turnover  Total Debt To Equity        ROA  \\\n",
       "ID                                                                     \n",
       "0      2.064582           22.240825           1325.517988   8.883952   \n",
       "0      1.842956           21.072092           1347.030183  12.324957   \n",
       "0      1.884968           21.077957           1345.780617  14.640880   \n",
       "\n",
       "          ROE  Gross Profit Margin  ...  Change in Equity to Fixed Assets  \\\n",
       "ID                                  ...                                     \n",
       "0   33.958244            25.273754  ...                               NaN   \n",
       "0   44.042406            27.514299  ...                        -23.494239   \n",
       "0   48.929598            28.984562  ...                          1.892533   \n",
       "\n",
       "    Change in Sales to Total Assets  Change in EBIT to revenue  \\\n",
       "ID                                                               \n",
       "0                               NaN                        NaN   \n",
       "0                         -6.434533                  29.515626   \n",
       "0                         -1.163430                  12.278989   \n",
       "\n",
       "    Change in Profit margin  Change in Sales to Inventory  \\\n",
       "ID                                                          \n",
       "0                       NaN                           NaN   \n",
       "0                 19.434621                     -0.686169   \n",
       "0                 19.276009                     -1.082365   \n",
       "\n",
       "    Change in Sales to Working capital  Change in R&D to Revenue  \\\n",
       "ID                                                                 \n",
       "0                                  NaN                       NaN   \n",
       "0                            52.633164                  1.116427   \n",
       "0                            -4.937915                 -6.448939   \n",
       "\n",
       "    Change in working cap to Assets  Change in Operating Income or Losses  \\\n",
       "ID                                                                          \n",
       "0                               NaN                                   NaN   \n",
       "0                         -3.893287                             21.181899   \n",
       "0                          0.917992                             10.972702   \n",
       "\n",
       "    Change in EBITDA Margin  \n",
       "ID                           \n",
       "0                       NaN  \n",
       "0                 28.212400  \n",
       "0                  2.175614  \n",
       "\n",
       "[3 rows x 51 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin by enriching our data with some additional columns.  In a typical machine learning workflow, the majority of the effort is usually dedicated to data cleaning and data preparation.  In order for us to run SVM successfully, we need to do a lot of the necessary work before we can actually feed the data into the model.  To enhance the data, we follow the below steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Enhance data:\n",
    "- Change in Earnings per share : (Current Period EPS - Prior Period EPS) \n",
    "- Assign 1 to positive change in EPS and 0 to negative change\n",
    "- Shift data index by -1: we will be using current financial data to predict future change in earnings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create binary column of positive and negative earnings changes\n",
    "data['binary change'] = [1 if row['change in EPS'] > 0 else 0 for _,row in data.iterrows()]\n",
    "\n",
    "# Shift date index by -1 so we are predicting future changes: 1 or 0\n",
    "data['Future change'] = data['binary change'].shift(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPS</th>\n",
       "      <th>change in EPS</th>\n",
       "      <th>Future change</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.044</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.424</td>\n",
       "      <td>0.38</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.934</td>\n",
       "      <td>0.51</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.084</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.064</td>\n",
       "      <td>-1.02</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.654</td>\n",
       "      <td>-1.41</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      EPS  change in EPS  Future change\n",
       "ID                                     \n",
       "0   3.044            NaN            1.0\n",
       "0   3.424           0.38            1.0\n",
       "0   3.934           0.51            1.0\n",
       "0   4.084           0.15            0.0\n",
       "0   3.064          -1.02            0.0\n",
       "0   1.654          -1.41            1.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Goal is to anticipate the sign of future earnings change from the financial data of the current quarter.\n",
    "# If the future earnings changes is + , we assign 1, otherwise 0,  to Future change value of the current quarter\n",
    "data[['EPS','change in EPS','Future change']].head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using pandas describe function to examine our data, you can see there are a number of columns that have negative and positive infinity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPS</th>\n",
       "      <th>change in EPS</th>\n",
       "      <th>Account Receivable Turnover</th>\n",
       "      <th>Current Ratio</th>\n",
       "      <th>Quick Ratio</th>\n",
       "      <th>Inventory Turnover</th>\n",
       "      <th>Total Debt To Equity</th>\n",
       "      <th>ROA</th>\n",
       "      <th>ROE</th>\n",
       "      <th>Gross Profit Margin</th>\n",
       "      <th>...</th>\n",
       "      <th>Change in EBIT to revenue</th>\n",
       "      <th>Change in Profit margin</th>\n",
       "      <th>Change in Sales to Inventory</th>\n",
       "      <th>Change in Sales to Working capital</th>\n",
       "      <th>Change in R&amp;D to Revenue</th>\n",
       "      <th>Change in working cap to Assets</th>\n",
       "      <th>Change in Operating Income or Losses</th>\n",
       "      <th>Change in EBITDA Margin</th>\n",
       "      <th>binary change</th>\n",
       "      <th>Future change</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4495.000000</td>\n",
       "      <td>3995.000000</td>\n",
       "      <td>3751.000000</td>\n",
       "      <td>3842.000000</td>\n",
       "      <td>3842.000000</td>\n",
       "      <td>2906.000000</td>\n",
       "      <td>4278.000000</td>\n",
       "      <td>4445.000000</td>\n",
       "      <td>4149.000000</td>\n",
       "      <td>3575.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>3.868000e+03</td>\n",
       "      <td>4.025000e+03</td>\n",
       "      <td>2736.000000</td>\n",
       "      <td>3410.000000</td>\n",
       "      <td>1287.000000</td>\n",
       "      <td>3388.000000</td>\n",
       "      <td>4.018000e+03</td>\n",
       "      <td>3721.000000</td>\n",
       "      <td>4545.000000</td>\n",
       "      <td>4544.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2.526763</td>\n",
       "      <td>0.048794</td>\n",
       "      <td>inf</td>\n",
       "      <td>2.991957</td>\n",
       "      <td>2.393492</td>\n",
       "      <td>24.179290</td>\n",
       "      <td>1422.670780</td>\n",
       "      <td>6.090243</td>\n",
       "      <td>29.921766</td>\n",
       "      <td>55.445803</td>\n",
       "      <td>...</td>\n",
       "      <td>inf</td>\n",
       "      <td>3.406696e+02</td>\n",
       "      <td>2.154707</td>\n",
       "      <td>54.832803</td>\n",
       "      <td>inf</td>\n",
       "      <td>0.340510</td>\n",
       "      <td>inf</td>\n",
       "      <td>-36.992755</td>\n",
       "      <td>0.467987</td>\n",
       "      <td>0.468090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.903321</td>\n",
       "      <td>2.803885</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.374618</td>\n",
       "      <td>1.099318</td>\n",
       "      <td>26.750252</td>\n",
       "      <td>888.674995</td>\n",
       "      <td>6.201164</td>\n",
       "      <td>36.878775</td>\n",
       "      <td>53.449220</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.887529e+04</td>\n",
       "      <td>38.520377</td>\n",
       "      <td>557.009442</td>\n",
       "      <td>NaN</td>\n",
       "      <td>594.324837</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1739.997723</td>\n",
       "      <td>0.499029</td>\n",
       "      <td>0.499036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-37.796000</td>\n",
       "      <td>-45.080000</td>\n",
       "      <td>12.512706</td>\n",
       "      <td>1.380189</td>\n",
       "      <td>1.251109</td>\n",
       "      <td>12.341636</td>\n",
       "      <td>1234.000000</td>\n",
       "      <td>-82.358224</td>\n",
       "      <td>-267.660000</td>\n",
       "      <td>-2003.262946</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.222895e+05</td>\n",
       "      <td>-4.041310e+04</td>\n",
       "      <td>-1225.694465</td>\n",
       "      <td>-970.974658</td>\n",
       "      <td>-100.000000</td>\n",
       "      <td>-6485.354365</td>\n",
       "      <td>-1.945754e+04</td>\n",
       "      <td>-102661.762624</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.624000</td>\n",
       "      <td>-0.310000</td>\n",
       "      <td>16.848874</td>\n",
       "      <td>2.226506</td>\n",
       "      <td>1.761880</td>\n",
       "      <td>15.088645</td>\n",
       "      <td>1283.288269</td>\n",
       "      <td>2.823914</td>\n",
       "      <td>18.352548</td>\n",
       "      <td>40.309807</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.498755e+01</td>\n",
       "      <td>-3.903804e+01</td>\n",
       "      <td>-10.095712</td>\n",
       "      <td>-20.948636</td>\n",
       "      <td>-7.500100</td>\n",
       "      <td>-10.985229</td>\n",
       "      <td>-3.016442e+01</td>\n",
       "      <td>-13.991613</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2.134000</td>\n",
       "      <td>0.035000</td>\n",
       "      <td>18.742693</td>\n",
       "      <td>2.619922</td>\n",
       "      <td>2.101069</td>\n",
       "      <td>17.086972</td>\n",
       "      <td>1325.637636</td>\n",
       "      <td>5.199796</td>\n",
       "      <td>23.269783</td>\n",
       "      <td>54.984085</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.277005e+00</td>\n",
       "      <td>-3.455357e+00</td>\n",
       "      <td>0.094533</td>\n",
       "      <td>-0.696431</td>\n",
       "      <td>-0.765746</td>\n",
       "      <td>-1.484532</td>\n",
       "      <td>-2.751739e+00</td>\n",
       "      <td>0.213443</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.864000</td>\n",
       "      <td>0.386667</td>\n",
       "      <td>23.213704</td>\n",
       "      <td>3.250817</td>\n",
       "      <td>2.601631</td>\n",
       "      <td>21.818833</td>\n",
       "      <td>1399.547976</td>\n",
       "      <td>8.807839</td>\n",
       "      <td>33.320514</td>\n",
       "      <td>74.795379</td>\n",
       "      <td>...</td>\n",
       "      <td>1.696780e+01</td>\n",
       "      <td>2.444517e+01</td>\n",
       "      <td>11.251677</td>\n",
       "      <td>21.098892</td>\n",
       "      <td>7.577819</td>\n",
       "      <td>7.065427</td>\n",
       "      <td>2.027000e+01</td>\n",
       "      <td>11.807783</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>83.314000</td>\n",
       "      <td>56.758548</td>\n",
       "      <td>inf</td>\n",
       "      <td>16.903861</td>\n",
       "      <td>12.961896</td>\n",
       "      <td>404.276095</td>\n",
       "      <td>44454.000000</td>\n",
       "      <td>52.774513</td>\n",
       "      <td>640.483713</td>\n",
       "      <td>109.258971</td>\n",
       "      <td>...</td>\n",
       "      <td>inf</td>\n",
       "      <td>1.185553e+06</td>\n",
       "      <td>550.720288</td>\n",
       "      <td>23774.482579</td>\n",
       "      <td>inf</td>\n",
       "      <td>31651.342509</td>\n",
       "      <td>inf</td>\n",
       "      <td>18111.658087</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 53 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               EPS  change in EPS  Account Receivable Turnover  Current Ratio  \\\n",
       "count  4495.000000    3995.000000                  3751.000000    3842.000000   \n",
       "mean      2.526763       0.048794                          inf       2.991957   \n",
       "std       3.903321       2.803885                          NaN       1.374618   \n",
       "min     -37.796000     -45.080000                    12.512706       1.380189   \n",
       "25%       1.624000      -0.310000                    16.848874       2.226506   \n",
       "50%       2.134000       0.035000                    18.742693       2.619922   \n",
       "75%       2.864000       0.386667                    23.213704       3.250817   \n",
       "max      83.314000      56.758548                          inf      16.903861   \n",
       "\n",
       "       Quick Ratio  Inventory Turnover  Total Debt To Equity          ROA  \\\n",
       "count  3842.000000         2906.000000           4278.000000  4445.000000   \n",
       "mean      2.393492           24.179290           1422.670780     6.090243   \n",
       "std       1.099318           26.750252            888.674995     6.201164   \n",
       "min       1.251109           12.341636           1234.000000   -82.358224   \n",
       "25%       1.761880           15.088645           1283.288269     2.823914   \n",
       "50%       2.101069           17.086972           1325.637636     5.199796   \n",
       "75%       2.601631           21.818833           1399.547976     8.807839   \n",
       "max      12.961896          404.276095          44454.000000    52.774513   \n",
       "\n",
       "               ROE  Gross Profit Margin  ...  Change in EBIT to revenue  \\\n",
       "count  4149.000000          3575.000000  ...               3.868000e+03   \n",
       "mean     29.921766            55.445803  ...                        inf   \n",
       "std      36.878775            53.449220  ...                        NaN   \n",
       "min    -267.660000         -2003.262946  ...              -2.222895e+05   \n",
       "25%      18.352548            40.309807  ...              -2.498755e+01   \n",
       "50%      23.269783            54.984085  ...              -1.277005e+00   \n",
       "75%      33.320514            74.795379  ...               1.696780e+01   \n",
       "max     640.483713           109.258971  ...                        inf   \n",
       "\n",
       "       Change in Profit margin  Change in Sales to Inventory  \\\n",
       "count             4.025000e+03                   2736.000000   \n",
       "mean              3.406696e+02                      2.154707   \n",
       "std               1.887529e+04                     38.520377   \n",
       "min              -4.041310e+04                  -1225.694465   \n",
       "25%              -3.903804e+01                    -10.095712   \n",
       "50%              -3.455357e+00                      0.094533   \n",
       "75%               2.444517e+01                     11.251677   \n",
       "max               1.185553e+06                    550.720288   \n",
       "\n",
       "       Change in Sales to Working capital  Change in R&D to Revenue  \\\n",
       "count                         3410.000000               1287.000000   \n",
       "mean                            54.832803                       inf   \n",
       "std                            557.009442                       NaN   \n",
       "min                           -970.974658               -100.000000   \n",
       "25%                            -20.948636                 -7.500100   \n",
       "50%                             -0.696431                 -0.765746   \n",
       "75%                             21.098892                  7.577819   \n",
       "max                          23774.482579                       inf   \n",
       "\n",
       "       Change in working cap to Assets  Change in Operating Income or Losses  \\\n",
       "count                      3388.000000                          4.018000e+03   \n",
       "mean                          0.340510                                   inf   \n",
       "std                         594.324837                                   NaN   \n",
       "min                       -6485.354365                         -1.945754e+04   \n",
       "25%                         -10.985229                         -3.016442e+01   \n",
       "50%                          -1.484532                         -2.751739e+00   \n",
       "75%                           7.065427                          2.027000e+01   \n",
       "max                       31651.342509                                   inf   \n",
       "\n",
       "       Change in EBITDA Margin  binary change  Future change  \n",
       "count              3721.000000    4545.000000    4544.000000  \n",
       "mean                -36.992755       0.467987       0.468090  \n",
       "std                1739.997723       0.499029       0.499036  \n",
       "min             -102661.762624       0.000000       0.000000  \n",
       "25%                 -13.991613       0.000000       0.000000  \n",
       "50%                   0.213443       0.000000       0.000000  \n",
       "75%                  11.807783       1.000000       1.000000  \n",
       "max               18111.658087       1.000000       1.000000  \n",
       "\n",
       "[8 rows x 53 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Examine data \n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will replace negative and positive infinity with NaN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace infinity with nan\n",
    "data = data.replace([np.inf, -np.inf], np.nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also drop the rows where the change in earnings per share is NaN.  We do this because we are trying to predict the change in earnings, so rows with NaN, or missing values, would not be useful information in our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Drop rows where change in EPS is nan: they are no use to us \n",
    "data = data.dropna(subset = ['change in EPS', 'Future change'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are also going to drop three columns, EPS, change in EPS, and binary change.  We no longer need these columns to continue examining the missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We no longer need these columns\n",
    "data = data.drop(columns = ['EPS','change in EPS','binary change'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see almost every column, other than future change, has some percentage of missing values and some columns have a substantial amount of missing values.  We have to deal with these missing values before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percent of missing values per column:\n",
      " Account Receivable Turnover                18.5\n",
      "Current Ratio                              15.3\n",
      "Quick Ratio                                15.3\n",
      "Inventory Turnover                         35.8\n",
      "Total Debt To Equity                        5.6\n",
      "ROA                                         1.5\n",
      "ROE                                         8.1\n",
      "Gross Profit Margin                        21.1\n",
      "Accounts Receivable Turnover               18.5\n",
      "Inventory to Sales                         16.6\n",
      "LT Debt to Total Equity                     4.9\n",
      "Sales to Total Assets                       0.2\n",
      "EBIT to revenue                             4.0\n",
      "Profit margin                               0.1\n",
      "Sales to Cash                               0.2\n",
      "Sales to Inventory                         32.1\n",
      "Sales to Working capital                   15.3\n",
      "Sales to Dep Fixed assets                  45.1\n",
      "Working capital to total Asset             15.3\n",
      "Operating Income to Total Assets            0.2\n",
      "Trailing 12M EBITDA Margin                  7.7\n",
      "Div as % of CF                              0.5\n",
      "change in Depreciation and Amortization     1.4\n",
      "change in Inventories                      32.2\n",
      "change in Inventory Turnover               36.3\n",
      "change in R&D Expense                      68.0\n",
      "change Total Assets                         0.2\n",
      "change in Long Term Debt                    1.1\n",
      "change in Short Term Debt                   6.0\n",
      "change in Revenue                           0.0\n",
      "change in Current Ratio                    15.4\n",
      "change in Quick Ratio                      15.4\n",
      "change in Tot Debt to Common Equity         6.8\n",
      "change in Gross Margin                     21.1\n",
      "Changes in Working Capital                 15.6\n",
      "Change in Inventory to Sales               33.6\n",
      "Change in Dep Amort Expense                 1.4\n",
      "Change in CAPAX to Assets                   3.8\n",
      "Change in LTD to Equity                     5.2\n",
      "Change in Equity to Fixed Assets           49.2\n",
      "Change in Sales to Total Assets             0.2\n",
      "Change in EBIT to revenue                   4.0\n",
      "Change in Profit margin                     0.1\n",
      "Change in Sales to Inventory               32.3\n",
      "Change in Sales to Working capital         15.4\n",
      "Change in R&D to Revenue                   68.0\n",
      "Change in working cap to Assets            16.0\n",
      "Change in Operating Income or Losses        0.3\n",
      "Change in EBITDA Margin                     7.7\n",
      "Future change                               0.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Examine missing data\n",
    "missing_column_data = 100*(data.isnull().sum() / data.shape[0]).round(3)\n",
    "print('Percent of missing values per column:\\n', missing_column_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Real world data often has missing values which require careful attention.  The handling of missing values is very important during the pre-processing step because many machine learning algorithms do not work with missing data.  There are two general ways of thinking about how to handle missing data.  One way is to delete the rows with the missing data, but we risk losing valuable information doing this.  The alternative is to try to compute the missing values using an array of different methods like mean or median imputation, neural networks, or Multiple Imputation by Chained Equations (MICE).\n",
    "\n",
    "In this exercise, we will drop columns that have more than 30% of data missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Inventory Turnover                  35.8\n",
       "Sales to Inventory                  32.1\n",
       "Sales to Dep Fixed assets           45.1\n",
       "change in Inventories               32.2\n",
       "change in Inventory Turnover        36.3\n",
       "change in R&D Expense               68.0\n",
       "Change in Inventory to Sales        33.6\n",
       "Change in Equity to Fixed Assets    49.2\n",
       "Change in Sales to Inventory        32.3\n",
       "Change in R&D to Revenue            68.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop 10 columns that have more than 30% of data missing\n",
    "columns_to_drop = missing_column_data[missing_column_data > 30]\n",
    "columns_to_drop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will result in us dropping ten columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New Dataframe shape : (3994, 40)\n"
     ]
    }
   ],
   "source": [
    "# Number of columns dropped, 10 \n",
    "data = data.drop(columns = list(columns_to_drop.index))\n",
    "print( f'New Dataframe shape : {data.shape}' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's continue with pre-processing our data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocess data:\n",
    "- Handle remaining missing values\n",
    "- Minimize influence of outliers by performing Winsorization\n",
    "- Standardize data \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Handle remaining missing data by replacing NaN by mean of the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep in mind that this is a naive way to handle missing values. \n",
    "# This method can cause data leakage and does not factor the covariance between features.\n",
    "# For more robust methods,take a look at MICE and KNN\n",
    "\n",
    "for col in data.columns:\n",
    "    data[col].fillna(data[col].mean(), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percent of missing values per column:\n",
      " Account Receivable Turnover                0.0\n",
      "Current Ratio                              0.0\n",
      "Quick Ratio                                0.0\n",
      "Total Debt To Equity                       0.0\n",
      "ROA                                        0.0\n",
      "ROE                                        0.0\n",
      "Gross Profit Margin                        0.0\n",
      "Accounts Receivable Turnover               0.0\n",
      "Inventory to Sales                         0.0\n",
      "LT Debt to Total Equity                    0.0\n",
      "Sales to Total Assets                      0.0\n",
      "EBIT to revenue                            0.0\n",
      "Profit margin                              0.0\n",
      "Sales to Cash                              0.0\n",
      "Sales to Working capital                   0.0\n",
      "Working capital to total Asset             0.0\n",
      "Operating Income to Total Assets           0.0\n",
      "Trailing 12M EBITDA Margin                 0.0\n",
      "Div as % of CF                             0.0\n",
      "change in Depreciation and Amortization    0.0\n",
      "change Total Assets                        0.0\n",
      "change in Long Term Debt                   0.0\n",
      "change in Short Term Debt                  0.0\n",
      "change in Revenue                          0.0\n",
      "change in Current Ratio                    0.0\n",
      "change in Quick Ratio                      0.0\n",
      "change in Tot Debt to Common Equity        0.0\n",
      "change in Gross Margin                     0.0\n",
      "Changes in Working Capital                 0.0\n",
      "Change in Dep Amort Expense                0.0\n",
      "Change in CAPAX to Assets                  0.0\n",
      "Change in LTD to Equity                    0.0\n",
      "Change in Sales to Total Assets            0.0\n",
      "Change in EBIT to revenue                  0.0\n",
      "Change in Profit margin                    0.0\n",
      "Change in Sales to Working capital         0.0\n",
      "Change in working cap to Assets            0.0\n",
      "Change in Operating Income or Losses       0.0\n",
      "Change in EBITDA Margin                    0.0\n",
      "Future change                              0.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Check for missing values\n",
    "missing_column_data = 100*(data.isnull().sum()/ data.shape[0]).round(3)\n",
    "print('Percent of missing values per column:\\n',missing_column_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we proceed further, we need to split the data into train and test.  Splitting data into train and test is absolutely necessary in machine learning to avoid overfitting.  It allows us to see how good our model really is and how well it performs on the new data we feed it.  We train the model on the training data and then make a prediction using the model that we learned in the training phase.  The prediction is made on the unlabeled test data.\n",
    "\n",
    "Here we split the data into train and test by allocating 80% of the data to train and 20% of the data to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we need to split our data into train and test. \n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Independent values/features\n",
    "X = data.iloc[:,:-1].values\n",
    "# Dependent values\n",
    "y = data.iloc[:,-1].values\n",
    "\n",
    "# Create test and train data sets, split data randomly into 20% test and 80% train\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to winsorize the data to limit the influence of the extreme values, typically by setting all outliers to a specified percentile of data.  Notice how we are winsorizing train data and test data separately.  If you winsorize all of your data together first and then partition it later into training and testing afterwards, you are allowing future data (i.e. test data) to influence your cutoff values.  Since you won't know what the future is when you use your model, you cannot use data manipulation affected by your future test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import mstats\n",
    "# Winsorize top 1% and bottom 1% of points. \n",
    "\n",
    "# Apply on X_train and X_test separately\n",
    "X_train = mstats.winsorize(X_train, limits = [0.01, 0.01])\n",
    "X_test = mstats.winsorize(X_test, limits = [0.01, 0.01])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one last thing that we have to do before we train the algorithm and that is to standardize the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$z=(x-mean) /  Standard Deviation$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standardization of a dataset is a common requirement for many machine learning estimators.  The reason for this is that these algorithms may not behave well if the individual features are not standard normally distributed data (e.g. Gaussian with 0 mean and unit variance).  This means there should be a mean of zero and unit variance.\n",
    "\n",
    "For instance many elements used in the objective function of a machine learning algorithm (such as the RBF kernel of Support-vector Machines (SVM) or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order.  If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standardize features by removing the mean and scaling to unit variance.\n",
    "\n",
    "# IMPORTANT: During testing, it is important to construct the test feature vectors using the means and standard deviations saved from\n",
    "# the training data, rather than computing it from the test data. You must scale your test inputs using the saved means\n",
    "# and standard deviations, prior to sending them to your SVM library for classification.\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "sc = StandardScaler()\n",
    "\n",
    "# Fit to training data and then transform it\n",
    "X_train = sc.fit_transform(X_train)\n",
    "# Perform standardization on testing data using mu and sigma from training data\n",
    "X_test = sc.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what are the advantages of support-vector machines?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Source: scikit-learn](https://scikit-learn.org/stable/modules/svm.html) <br>\n",
    "\n",
    "### SVM\n",
    "\n",
    "**Advantages:**\n",
    "* Effective in high dimensional spaces.\n",
    "* Still effective in cases where the number of dimensions is greater than the number of samples.\n",
    "* Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.\n",
    "* Versatile: different Kernel functions can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels.\n",
    "\n",
    "**Disadvantages:**\n",
    "* If the number of features is much greater than the number of samples, it is crucial to avoid over-fitting in choosing Kernel functions and the regularization term.\n",
    "* It also doesn’t perform very well when the data set has more noise (i.e. when target classes are overlapping).\n",
    "* SVMs do not directly provide probability estimates, these are calculated using an expensive five-fold cross-validation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/svm.jpg'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train SVM on our data.  We will first use default parameters for C, kernel, and gamma."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Support Vector Classification(C)\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "# Initialize svm, rbf is a default kernel\n",
    "classifier_rbf = SVC(C = 1, kernel = 'rbf', gamma = 'auto', random_state = 0)\n",
    "\n",
    "# Fit the model on training data\n",
    "classifier_rbf.fit(X_train, y_train)\n",
    "\n",
    "# Make a prediction on testing data\n",
    "y_pred_rbf = classifier_rbf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also check the model's accuracy score and classification report.  In the classification report, precision, recall, and f1-scores are given.  Precision quantifies the number of positive class predictions that actually belong to the positive class.  Recall quantifies the number of positive class predictions made out of all positive examples in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy with RBF: 0.58\n"
     ]
    }
   ],
   "source": [
    "# Import accuracy score\n",
    "from sklearn.metrics import accuracy_score\n",
    "ac_rbf = accuracy_score(y_test, y_pred_rbf)\n",
    "print('Accuracy with RBF: {:.2f}'.format(ac_rbf))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.57      0.77      0.66       413\n",
      "         1.0       0.61      0.38      0.47       386\n",
      "\n",
      "    accuracy                           0.58       799\n",
      "   macro avg       0.59      0.58      0.56       799\n",
      "weighted avg       0.59      0.58      0.57       799\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Precision and recall\n",
    "from sklearn.metrics import classification_report\n",
    "result = classification_report(y_test, y_pred_rbf)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if we can improve this algorithm by tuning some of the hyperparameters.  So what are the hyperparameters?\n",
    "\n",
    "In machine learning, hyperparameters are those parameters whose values are used to control the learning process.  Their configuration is external to the model and the tuning process usually involves discovering hyperparameters that result in the model making the most skillful predictions.  Hyperparameters are often used in the process to help estimate the model parameters.  Let's look at the three most important hyperparameters in our example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameters:\n",
    "- Kernel - transforms the data into a required form (i.e. dimension) so the data can be separated. RBF is useful for non-linear hyperplane in higher dimensions\n",
    "  and computes the separation line in the higher dimension. In some of the applications, it is suggested to use a more complex kernel to separate the classes that are curved or nonlinear.\n",
    "- Regularization, C - penalty parameter, which represents misclassification or error. It tells the SVM optimization how much error is bearable. Small C results in a small-margin hyperplane while large C results in a large margin hyperplane.\n",
    "- Gamma - defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’. The gamma parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors. Higher values of gamma will exactly fit the training dataset, which can cause overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy with Linear: 0.57\n"
     ]
    }
   ],
   "source": [
    "# Default C = 1, let's change kernel to linear\n",
    "classifier_lin = SVC(C = 1, kernel = 'linear',gamma = 'auto',random_state=0)\n",
    "\n",
    "# Fit the model on training data\n",
    "classifier_lin.fit(X_train, y_train)\n",
    "\n",
    "# Make a prediction on testing data\n",
    "y_pred_lin = classifier_lin.predict(X_test)\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "ac_lin = accuracy_score(y_test, y_pred_lin)\n",
    "print('Accuracy with Linear: {:.2f}'.format(ac_lin))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding optimal hyperparameters is a tedious task to solve, but it can be done by trying various combinations of hyperparameters to see which parameters work best."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can we speed up our SVM algorithm?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Principal Component Analysis (PCA)\n",
    "- Common way to speed up machine learning algorithms\n",
    "- Large number of features in the dataset can affect both the training times and accuracy of the model\n",
    "- PCA is a statistical technique that reduces number of features to those that capture maximum information about the dataset\n",
    "- Features are selected on the basis of their variance - higher the variance, more information that component conveys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# keep 95% of variance\n",
    "pca = PCA(0.95)\n",
    "X_train_pca = pca.fit_transform(X_train)\n",
    "X_test_pca = pca.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Components that explain 95% of variance in our dataset\n",
    "explained_variance = pca.explained_variance_ratio_\n",
    "# 28 features explain 95% of variance, down from original 40\n",
    "len(explained_variance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Able to achieve similar accuracy but with only 28 features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.57\n"
     ]
    }
   ],
   "source": [
    "classifier = SVC(C = 1, kernel='rbf',gamma = 'auto',random_state=0)\n",
    "\n",
    "classifier.fit(X_train_pca, y_train)\n",
    "y_pred = classifier.predict(X_test_pca)\n",
    "ac = accuracy_score(y_test, y_pred)\n",
    "print('Accuracy: {:.2f}'.format(ac))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen, PCA can be a powerful tool in your machine learning implementation.  Not only have we cut 12 features from our dataset, we were also able to improve the speed of the algorithm while achieving a similar prediction accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "SVMs can be a powerful tool in your machine learning arsenal.  SVM provides a very useful technique when using the kernel.  Within some classification problems, there is a strong assumption that data samples are linearly separable.  But with the introduction of the kernel, the input data can be converted into high dimensional data avoiding the need for this assumption.  So we no longer have to worry about the data needing to be linearly separable because it can be converted into high dimensional data.  SVMs generally do not suffer from overfitting and perform very well when there is a clear indication of separation between classes.  Also, SVMs can be used when the total number of samples is less than the number of features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Resources\n",
    "\n",
    "#### Helpful Blog Posts\n",
    "Machine Learning for Investing: https://hdonnelly6.medium.com/list/machine-learning-for-investing-7f2690bb1826\n",
    "\n",
    "#### Python Libraries\n",
    "\n",
    "Scikit train_test_split:\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html\n",
    "\n",
    "Scikit SVM:\n",
    "https://scikit-learn.org/stable/modules/svm.html\n",
    "    \n",
    "PCA:\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html\n",
    "\n",
    "Missing values imputation\n",
    "https://scikit-learn.org/stable/modules/impute.html\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}