Day1/CorrallingUnrulyData.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Wranglin' – Corralling Unruly Data\n",
    "One bit at a time\n",
    "=====\n",
    "\n",
    "**Abridged Version 0.2.5**\n",
    "\n",
    "By B Scott, Northwestern/CIERA\n",
    "adapted from Version by A Miller\n",
    "May 2024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "For this exercise you will need a five different text files. They have been compiled into a tarball that you should [download](https://arch.library.northwestern.edu/downloads/8g84mm66j?locale=en) and unpack in the same directory as this notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If data are constants, and constants don't change, then we should probably be sure that our data storage solutions do not alter the data in any way.\n",
    "\n",
    "Within the data science community, the python pandas package is particularly popular for reading, writing, and manipulating data (there will be several examples this week).\n",
    "\n",
    "The pandas docs state the read_csv() method is the workhorse function for reading text files. So, does pandas \"maintain the constant nature of data\"?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Problem 1a**\n",
    "\n",
    "Create a numpy array, called nums, of length 10000 filled with random numbers. Create a pandas Series object, called s, based on that array, and then write the Series to a file called tmp.txt using the to_csv() method.\n",
    "\n",
    "Hint - you'll need to name the Series and add the header=True option to the to_csv() call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nums = # complete\n",
    "s = # complete\n",
    "# complete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Problem 1b**\n",
    "\n",
    "Using the pandas read_csv() method, read in the data to a new variable, called s_read. Do you expect s_read and nums to be the same? Check whether or not your expectations are correct.\n",
    "\n",
    "Hint - take the sum of the difference not equal to zero to identify if any elements are not the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s_read = # complete\n",
    "\n",
    "# complete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, it turns out that ∼23% of the time, pandas does not in fact read in the same number that it wrote to disk.\n",
    "\n",
    "The truth is that these differences are quite small (see next slide), but there are many mathematical operations (e.g., subtraction of very similar numbers) that may lead these tiny differences to compound over time such that your data are not, in fact, constant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.max(np.abs(nums - s_read['nums'].values)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, what is going on?\n",
    "\n",
    "Sometimes, when you convert a number to ASCII (i.e. text) format, there is some precision that is lost in that conversion.\n",
    "\n",
    "How do you avoid this?\n",
    "\n",
    "One way is to directly write your files in binary. To do so has serveral advantages: it is possible to reproduce byte level accuracy, and, binary storage is almost always more efficient than text storage (the same number can be written in binary with less space than in ascii).\n",
    "\n",
    "The downside is that developing your own procedure to write data in binary is a pain, and it places strong constraints on where and how you can interact with the data once it has been written to disk.\n",
    "\n",
    "Fortuantely, we live in a world with pandas. All this hard work has been done for you, as pandas naturally interfaces with the hdf5 binary table format. (You may want to also take a look at pyTables)\n",
    "\n",
    "(Historically astronomers have used FITS files as a binary storage solution)\n",
    "\n",
    "**Problem 1c**\n",
    "\n",
    "Repeat your procedure from above, but instead of writing to a csv file, use the pandas to_hdf() and read_df() method to see if there are any differences in s and s_read.\n",
    "\n",
    "Hint - You will need to specify a name for the table that you have written to the hdf5 file in the call to to_hdf() as a required argument. Any string will do.\n",
    "\n",
    "Hint 2 - Use s_read.values instead of s_read['nums'].values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s. # complete\n",
    "s_read = # complete\n",
    "\n",
    "# complete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, if you are using pandas anyway, and if you aren't using pandas –– check it out!, then I strongly suggest removing csv files from your workflow to instead focus on binary hdf5 files. This requires typing the same number of characters, but it ensures byte level reproducibility.\n",
    "\n",
    "And reproducibiliy is the pillar upon which the scientific method is built.\n",
    "\n",
    "Is that the end of the story? ... No.\n",
    "\n",
    "In the previous example, I was being a little tricky in order to make a point. It is in fact possible to create reproducible csv files with pandas. By default, pandas sacrifices a little bit of precision in order to gain a lot more speed. If you want to ensure reproducibility then you can specify that the float_precision should be round_trip, meaning you get the same thing back after reading from a file that you wrote to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s.to_csv('tmp.txt', header=True, index=False)\n",
    "\n",
    "s_read = pd.read_csv('tmp.txt', float_precision='round_trip')\n",
    "\n",
    "sum(nums - s_read['nums'].values != 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So was all of this in service of a lie?\n",
    "\n",
    "No. What I said before remains true - text files do not guarantee byte level precision, and they take more space on disk. Text files have some advantages:\n",
    "\n",
    "anyone, anywhere, on any platform can easily manipulate text files\n",
    "text files can be easily inspected (and corrected) if necessary\n",
    "special packages are needed to read/write in binary\n",
    "binary files, which are not easily interpretable, are difficult to use in version control (and banned by some version control platforms)\n",
    "To summarize, here is my advice: think of binary as your (new?) default for storing data.\n",
    "\n",
    "But, as with all things, consider your audience: if you are sharing/working with people that won't be able to deal with binary data, or, you have an incredibly small amount of data, csv (or other text files) should be fine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Problem 2) The Sins of Our Predecessors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If at any point in your career you need to access archival infrared data, you will likely need to retrieve that information from the [NASA IPAC InfraRed Science Archive](https://irsa.ipac.caltech.edu). IRSA houses the data for every major NASA IR mission, and several ground-based missions as well (e.g., 2MASS, IRTF). Whether you are sudying brown dwarfs, explosive transients, solar system objects, star-formation, galaxy evolution, Milky Way dust and the resulting extinction of extragalactic observations, or quasars (and much more) the IR plays a critical role."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Given the importance of IR observations, it stands to reason that IRSA would provide data in a simple to read format for modern machines, such as comma separated values or FITS binary tables...\n",
    "\n",
    "Right?..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Right?...**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In fact, IRSA has created their own standard for storing data in a text file. The particulars of this format can be found in `irsa_catalog_WISE_iPTF14jg_search_results.tbl`, a file that is written in the standard IRSA format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "*shameless plug for Adam alert!* iPTF14jg is a [really strange star](https://arxiv.org/pdf/1901.10693.pdf) that exhibited a large outburst that we still don't totally understand. The associated data file includes [NEOWISE](https://neowise.ipac.caltech.edu/) observations of the mid-IR evolution of this outburst."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 2a**\n",
    "\n",
    "Using `pandas` read the data in the IRSA table file into a `DataFrame` object.\n",
    "\n",
    "*Hint 1* - you absolutely should look at the text file to develop a strategy to accomplish this goal.\n",
    "\n",
    "*Hint 2* - you may want to manipulate the text file so that it can more easily be read by `pandas`. **If you do this** be sure to copy the file to another name as you will want to leave the original intact. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Solution 1 - pure python solution with pandas\n",
    "\n",
    "with open('irsa_catalog_WISE_iPTF14jg_search_results.tbl') as f:\n",
    "    ll = f.readlines()\n",
    "    for linenum, l in enumerate(ll):\n",
    "        if l[0] == '|':\n",
    "            header = l.replace('|', ',').replace(' ', '')\n",
    "            header = list(header[1:-2].split(','))\n",
    "            break\n",
    "            \n",
    "irsa_tbl = pd.read_csv(\"irsa_catalog_WISE_iPTF14jg_search_results.tbl\", \n",
    "            skiprows=linenum+4, delim_whitespace=True, \n",
    "            header=None, names=header)\n",
    "irsa_tbl.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "That pure python solution is a bit annoying as it requires a for loop with a break, and specific knowledge about how IRSA tables handle data headers (hence the use of `linenum + 4` for `skiprows`). Alternatively, one could  manipulate the data file to read in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# solution 2 - edit the text file\n",
    "# !cp irsa_catalog_WISE_iPTF14jg_search_results.tbl tmp.tbl\n",
    "### delete lines 1-89, and 90-92\n",
    "### replace whitespace with commas (may require multiple operations)\n",
    "### replace '|' with commas\n",
    "### replace ',,' with single commas\n",
    "### replace ',\\n,' with '\\n'\n",
    "### delete the comma at the very beginning and very end of the file\n",
    "\n",
    "tedit_tbl = pd.read_csv('tmp.tbl')\n",
    "tedit_tbl.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "That truly wasn't all that better - as it required a bunch of clicks/text editor edits. (There are programs such as `sed` and `awk` that could be used to execute all the necessary edits from the command line, but that too is cumbersome and somewhat like the initial all `python` solution). \n",
    "\n",
    "If astronomers are creating data in a \"standard\" format, then it ought to be easy for other astronomers to access that data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fortunately, in this particular case, there is an easy solution - [`astropy Tables`](http://docs.astropy.org/en/stable/table/). \n",
    "\n",
    "IRSA tables are so commonly used throughout the community, that the folks at `astropy` have created a convenience method for all of us to read in tables created in that particular (unusual?) format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 2b**\n",
    "\n",
    "Use [`Table.read()`](http://docs.astropy.org/en/stable/api/astropy.table.Table.html#astropy.table.Table.read) to read in `irsa_catalog_WISE_iPTF14jg_search_results.tbl` to an `astropy Table` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from astropy.table import Table\n",
    "\n",
    "Table.read('irsa_catalog_WISE_iPTF14jg_search_results.tbl', format='ipac')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A benefit to using this method, as opposed to `pandas`, is that data typing and data units are naturally read from the IRSA table and included with the associated columns. Thus, if you are uncertain if some brightness measurement is in magnitudes or Janskys, the `astropy Table` can report on that information.\n",
    "\n",
    "Unfortunately, `astropy` does *not* know about every strange formating decision that every astronomer has made at some point in their lives (as we are about to see...) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Problem 3) The Sins of Our Journals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Unlike IRSA/IPAC, which uses a weird but nevertheless consistent format for data tables, data retrieved from Journal articles essentially follows no rules. In principle, tables in Journal articles are supposed to be provided in a machine readable format. In practice, as we are about to see, this is far from the case.\n",
    "\n",
    "For this particular wrangling case study we will focus on supernova light curves, a simple thing to report: time, filter, brightness, uncertainty on that brightness, that the community has nevertheless managed to mangle into some truly wild and difficult to parse forms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "(Sorry for the heavy emphasis on time-domain examples - I'm pulling straight from my own life today, but the issues described here are not perfectly addressed by any subfield within the astro umbrella)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is the LaTeX-formatted version of Table 4 from [Miller et al. 2011](https://iopscience.iop.org/article/10.1088/0004-637X/730/2/80/meta):\n",
    "\n",
    "<img style=\"display: block; margin-left: auto; margin-right: auto\" src=\"images/Miller11_tbl4.png\" width=\"350\" align=\"middle\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "That is a very simple table to interpret, no?\n",
    "\n",
    "Have a look at the [\"machine-readible\" file](https://iopscience.iop.org/0004-637X/730/2/80/suppdata/apj382770t4_ascii.txt?doi=10.1088/0004-637X/730/2/80) that ApJ provides for readers that might want to evaluate these photometric measurements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 3a** \n",
    "\n",
    "Read the ApJ version of Table 4 from from Miller et al. 2011 – called `Miller_et_al2011_table4.txt` – into either a `pandas DataFrame` or an `astropy Table`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# pure python solution with pandas\n",
    "\n",
    "tbl4 = pd.read_csv(, \n",
    "                   skiprows=, delim_whitespace=,\n",
    "                   skipfooter=, engine='python',\n",
    "                   names=[])\n",
    "tbl4.drop(columns=[], inplace=True)\n",
    "\n",
    "print(tbl4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "That wasn't too terrible. But what if we consider a more typical light curve table, where there are loads of missing data, such as Table 2 from [Foley et al. 2009](https://iopscience.iop.org/article/10.1088/0004-6256/138/2/376#aj309430t2):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Again, this table is straightforward to read, and it isn't hard to imagine how one could construct a machine-readable csv or other file from this information. But alas, this is not what is available from ApJ. So, we will need to figure out how to deal with both the missing data, \"...\", and the weird convention that many astronomers use where the uncertainties are (a) not reported in their own column, and (b) are not provided in the same units as the measurement itself. I can understand the former, but the later is somewhat baffling..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 3b** \n",
    "\n",
    "Read the ApJ version of Table 2 from from Foley et al. 2009 – called `Foley_et_al2009_table2.txt` – into either a `pandas DataFrame` or an `astropy Table`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# a (not terribly satisfying) pure python solution\n",
    "# read the file in, parse and write another file that plays nice with pandas\n",
    "\n",
    "with open('Foley_et_al2009_for_pd.csv','w') as fw:\n",
    "    print('JD,Bmag,Bmag_unc,Vmag,Vmag_unc,Rmag,Rmag_unc,Imag,Imag_unc,Unfiltmag,Unfiltmag_unc,Telescope',file=fw)\n",
    "    with open('Foley_et_al2009_table2.txt') as f:\n",
    "        ll = f.readlines()\n",
    "        for l in ll:\n",
    "            if l[0] == '2':\n",
    "                print_str = l.split()[0] + ','\n",
    "                for col in l.split()[1:]:\n",
    "                    if col == 'sdotsdotsdot':\n",
    "                        print_str += \n",
    "                    #complete \n",
    "                    else:\n",
    "                        print_str += '{},'.format(col)\n",
    "\n",
    "                print(print_str,file=fw)\n",
    "\n",
    "pd.read_csv('Foley_et_al2009_for_pd.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Okay - there is nothing elegant about that particular solution. But it works, and wranglin' ain't pretty. \n",
    "\n",
    "It is likely that you developed a solution that looks very different from this one, and that is fine. When data are provided in an unrulely format, the most important thing is to develop some method, any method, for converting the information into a useful format. Following whatever path you used above, it should now be easy to plot the light curve of SN 2008ha."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Problem 4) My Heart Will Go On\n",
    "\n",
    "Sometimes there is no difficultly whatsoever in reading in the data (as was the case in **Problems 2** and **3**), but instead the difficultly lies in wranglin' the data to be appropriate for the model that you are building.\n",
    "\n",
    "For the next problem we will work with the famous [Titanic survival](https://www.kaggle.com/c/titanic/data?) data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Briefly, [the Titantic](https://thefilmcricket.files.wordpress.com/2012/04/film-titanic_clar.jpg) is a [famous](https://wallpapercave.com/wp/jrF8rQK.jpg) historical ship that was thought to be unsinkable. **Spoiler alert** it hit an iceberg and sank. The data in the Titanic data set includes information about 891 passengers from the Titanic, as well as whether or not they survived. The aim of this data set is to build a machine learning model to predict which passengers survived and which did not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The features include: \n",
    "\n",
    "|Feature    | Description |\n",
    "|:---------:|:--------------------------------------:|\n",
    "|PassengerId| Running index that describes the individual passengers|\n",
    "|Pclass| A proxy for socio-economic status (1 = Upper class, 2 = Middle Class, 3 = Lower Class)|\n",
    "|Name| The passenger's name|\n",
    "|Sex | The passenger's sex|\n",
    "|Age | The passenger's age - note age's ending in 0.5 are estimated |\n",
    "|SibSp| The sum of the passenger's sibblings and spouces on board|\n",
    "|Parch| The sum of the passenger's parents and children on board|\n",
    "|Ticket| The ticket number for the passenger|\n",
    "|Fare| The price paid for the ticket by th passenger|\n",
    "|Cabin| The Cabin in which the passenger stayed|\n",
    "|Embarked| The point of Origin for the Passenger: C = Cherbourg, S = Southampton, Q = Queenstown|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "And of course, we are trying to predict:\n",
    "\n",
    "|Label    | Description |\n",
    "|:---------:|:--------------------------------------:|\n",
    "|Survived| 1 = yes; 0 = no|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4a**\n",
    "\n",
    "Read in the Titanic training data and create the `scikit-learn` standard `X` and `y` arrays to hold the features and the labels, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "titanic_df = pd.read_csv(, comment=)\n",
    "\n",
    "X = titanic_df[]\n",
    "y = titanic_df[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now that we have the data in the appropriate `X` and `y` arrays, estimate the accuracy with which a [K nearest neighbors](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) classification model can predict whether or not a passenger would survive the Titanic disaster. Use $k=10$ fold cross validation for the prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4b**\n",
    "\n",
    "Train a $k=7$ nearest neighbors machine learning model on the Titanic training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "knn_clf = KNeighborsClassifier(n_neighbors=7)\n",
    "knn_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note - that should have failed! And for good reason - recall that `kNN` models measure the Euclidean distance between all points within the feature space. So, when considering the sex of a passenger, what is the *numerical* distance between male and female? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In other words, we need to wrangle this data before we can run the machine learning model. \n",
    "\n",
    "Most of the features in this problem are non-numeric (i.e. we are dealing with categorical features), and therefore we need to figure out how to include them in the `kNN` model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The first step when wrangling for machine learning is to figure out if anything can be thrown away. We certainly want to avoid including any uninformative features in the model. \n",
    "\n",
    "*If you haven't already, now would be a good time to create a new cell and examine the contents of the csv*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "titanic_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4c** \n",
    "\n",
    "Are there any features that *obviously* will not help with this classification problem?\n",
    "\n",
    "*If you answer yes - ignore those features moving forward*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Solution 4c**\n",
    "\n",
    "*write your solution here*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Given that we have both categorical and numeric features, let's start with the numerical features and see how well they can predict survival on the Titanic.\n",
    "\n",
    "One note - for now we are going to exclude `Age`, because as you saw when you examined the data, there are some passengers that do not have any age information. This problem, known as \"missing data\" is one that we will deal with before the end of this problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4d**\n",
    "\n",
    "How accurately can the numeric features, `Pclass`, `SibSp`, `Parch`, and `Fare` predict survival on the Titanic? Use a $k = 7$ Nearest Neighbors model, and estimate the model accuracy using 10-fold cross validation. \n",
    "\n",
    "*Hint 1 - you'll want to redefine your features vector `X`*\n",
    "\n",
    "*Hint 2 - you may find [`cross_val_score`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) from `scikit-learn` helpful*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "X = # complete\n",
    "\n",
    "knn_clf = # complete\n",
    "\n",
    "cv_results = cross_val_score( # complete\n",
    "\n",
    "print('The accuracy from numeric features = {:.2f}%'.format(100*np.mean(cv_results)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "An accuracy of 68% isn't particularly inspiring. But, there's a lot of important information that we are excluding. As far as the Titanic is concerned, Kate and Leo taught us that [female passengers are far more likely to survive](https://qph.fs.quoracdn.net/main-qimg-93eb36091c7eec872b891fa51dc5722b), while [male passengers are not](http://hoycinema.abc.es/Media/201602/03/titanic-kate-dicaprio--644x362.jpg). So, if we can include gender in the model then we may be able to achieve more accurate predictions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4e**\n",
    "\n",
    "Create a new feature called `gender` that equals 1 for male passengers and 2 for female passengers. Add this feature to your dataframe, and include it in a `kNN` model with the other numeric features. Does the inclusion of this feature improve the 10-fold CV accuracy?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "gender = # complete\n",
    "# complete\n",
    "\n",
    "titanic_df['gender'] = gender\n",
    "\n",
    "X = # complete\n",
    "\n",
    "cv_results = cross_val_score( # complete\n",
    "\n",
    "print('The accuracy when including gender = {:.2f}%'.format(100*np.mean(cv_results)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A 14% improvement is pretty good! But, we can wrangle even more out of the gender feature. Recall that `kNN` models measure the Euclidean distance between sources, meaning the scale of each feature really matters. Given that the fare ranges from 0 up to 512.3292, the `kNN` model will see this feature as far more important than `gender`, for no other reason than the units that have been adopted. \n",
    "\n",
    "If women are far more likely to survive than men, then we want to be sure that gender is weighted at least the same as all the other features, which we can do with a minmax scaler. As a brief reminder - a minmax scaler scales all values of a feature to be between 0 and 1 by subtracting the minimum value of each feature and then dividing by the maximum minus the minimum. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4f**\n",
    "\n",
    "Scale all the features from the previous problem using a minmax scaler and evaluate the CV accuracy of the `kNN` model.\n",
    "\n",
    "*Hint - you may find [`MinMaxScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html) helpful*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "scaler = MinMaxScaler()\n",
    "scaler.fit(X)\n",
    "Xminmax = scaler.transform(X)\n",
    "\n",
    "knn_clf = # complete\n",
    "\n",
    "cv_results = cross_val_score( # complete\n",
    "\n",
    "print('The accuracy when scaling features = {:.2f}%'.format(100*np.mean(cv_results)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Scaling the features leads to further improvement!\n",
    "\n",
    "Now turn your attention to another categorical feature, `Embarked`, which includes the point of origin for each passenger and has three categories, `S`, `Q`, and `C`. We need to convert these values to a numeric representation for inclusion in the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4g**\n",
    "\n",
    "Convert the categorical feature `Embarked` to a numeric representation, and add it to the `titanic_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# following previous example, set C = 0, S = 1, Q = 2\n",
    "\n",
    "porigin = # complete\n",
    "# complete\n",
    "# complete\n",
    "# complete\n",
    "\n",
    "titanic_df['porigin'] = porigin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But wait! Does this actually make sense?\n",
    "\n",
    "Our \"numerification\" has now introduced order where there previously was none. We are effectively telling the model that Cherbourg and Queenstown are far apart (not in distance but in terms of the similarity of the passengers that boarded the ship in each location), while each are equally close to Southampton. Is there actually any evidence to support this conclusion? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "By definition categorical features do not have order (e.g., cat, dog, horse, elephant), and therefore we should not impose any when converting these features to numeric values for inclusion in our model. Instead, we should be creating a new set of binary features for every category within the feature set. Thus, `Embarked` will now need to be represented by 3 different features, where the feature `Queenstown` equals one for passengers that boarded there and zero for everyone else. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4h**\n",
    "\n",
    "Complete the function below that will automatically create binary arrays for a categorical feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def create_bin_cat_feats(feature_array):\n",
    "    categories = np.unique(feature_array)\n",
    "    feat_dict = {}\n",
    "    for cat in categories:\n",
    "        #complete\n",
    "    \n",
    "    return feat_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4i**\n",
    "\n",
    "Use the `create_bin_cat_feats` function to convert the `Embarked` and `Sex` categorical features to a numeric representation (yes we need to do this for `Sex` as well where we otherwise previously introduced order). Add these features to the `titanic_df` data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "gender_dict = \n",
    "porigin_dict = \n",
    "\n",
    "for feat in gender_dict.keys():\n",
    "    titanic_df[feat] = gender_dict[feat]\n",
    "    \n",
    "for feat in porigin_dict.keys():\n",
    "    titanic_df[feat] = porigin_dict[feat]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4j**\n",
    "\n",
    "Use the newly created `female`, `male`, `S`, `Q`, and `C` features in combination with the `Pclass`, `SibSp`, `Parch`, and `Fare` features to estimate the classification accuracy of a $k = 7$ nearest neighbors model with 10-fold cross validation.\n",
    "\n",
    "How does the addition of the point of origin feature affect the final model output?\n",
    "\n",
    "*Hint - don't forget to scale the features in the model*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "X = # complete\n",
    "\n",
    "scaler = MinMaxScaler()\n",
    "scaler.fit(X)\n",
    "Xminmax = scaler.transform(X)\n",
    "\n",
    "cv_results = cross_val_score( # complete\n",
    "\n",
    "print('The accuracy with categorical features = {:.2f}%'.format(100*np.mean(cv_results)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The last thing we'd like to add to the model is the `Age` feature. Unfortunately, for 177 passengers we do not have a reported value for their age. This is a standard issue when building models known as \"missing data\" and this happens in astronomy all the time (for example, LSST is going to observe millions of L and T dwarfs that are easily detected in the $y$-band, but which will not have a detection in $u$-band)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are several different strategies for dealing with missing data. The first and most straightforward is to simply remove observations with missing data (note - to simplify this example I already did this by removing the 2 passengers from the training set that did not have an entry for `Embarked`). \n",
    "\n",
    "This strategy is perfectly fine if only a few sources have missing information (2/891 for `Embarked` - and none of the test set sources are missing `Embarked`). If, however, a significant fraction are missing data, this strategy would remove a lot of useful data from the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you cannot remove the sources with missing data, then it is essential to ask the following question: \n",
    "\n",
    "Does the missing information have meaning? \n",
    "\n",
    "In the LSST L/T dwarf example, the lack of a $u$-band detection is meaningful: these stars are too faint for LSST. When this is the case, an indicator value (e.g., -999) allows the model to recognize the non-detection. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the Titanic data, the lack of age information is not meaningful. Simply put, there are some passengers that did not have recorded ages. We will now show this to be the case. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4k**\n",
    "\n",
    "Replace the unknown ages with a value of -999, and estimate the accuracy of the model via 10-fold cross validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "age_impute = # complete\n",
    "# complete\n",
    "\n",
    "titanic_df['age_impute'] = age_impute\n",
    "\n",
    "X = # complete\n",
    "\n",
    "scaler = MinMaxScaler()\n",
    "scaler.fit(X)\n",
    "Xminmax = scaler.transform(X)\n",
    "\n",
    "cv_results = cross_val_score( # complete\n",
    "\n",
    "print('The accuracy with -999 for missing ages = {:.2f}%'.format(100*np.mean(cv_results)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The accuracy of the model hasn't improved by adding the age information (even though we know children were more likely to survive than adults). \n",
    "\n",
    "Given that the missing ages don't have meaning, we need to develop alternative strategies for \"imputing\" the missing data. The most simple approach in this regard is to replace the missing values with the mean value of the feature distribution for sources that do have measurements (use the median if the distribution has significant outliers).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 4l**\n",
    "\n",
    "Replace the unknown ages with the mean age of passengers, and estiamte the accuracy of the model via 10-fold cross validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "age_impute = # complete\n",
    "# complete\n",
    "\n",
    "titanic_df['age_impute'] = age_impute\n",
    "\n",
    "X = # complete\n",
    "\n",
    "scaler = MinMaxScaler()\n",
    "scaler.fit(X)\n",
    "Xminmax = scaler.transform(X)\n",
    "\n",
    "cv_results = cross_val_score( # complete\n",
    "\n",
    "print('The accuracy with the mean for missing ages = {:.2f}%'.format(100*np.mean(cv_results)))"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "livereveal": {
   "height": 768,
   "scroll": true,
   "start_slideshow_at": "selected",
   "theme": "solarized",
   "width": 1024
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}