LoS_CoF_Module2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step - 2: Legacy of Slavery Certificates of Freedom Collection - Context Based Data Manipulation and Analysis - Part 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Height Feature \n",
    "This field is to indicate the height of the individual freed in feet and inches. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries used for data frame (table-like) operations, and numeric data structure operations\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code to import the csv saved from the previous step\n",
    "df = pd.read_csv(\"Datasets/LoS_Clean_Output_Mod1.csv\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          5' 3\"\n",
       "1          5' 3\"\n",
       "2           5'3\"\n",
       "3        5'7.75\"\n",
       "4         4'9.5\"\n",
       "          ...   \n",
       "23650    5'8.25\"\n",
       "23651       5'9\"\n",
       "23652     5'7.5\"\n",
       "23653       5'7\"\n",
       "23654       5'5\"\n",
       "Name: Height, Length: 23655, dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#code to pull the error above\n",
    "df[\"Height\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     22325\n",
       "unique      533\n",
       "top        5'4\"\n",
       "freq       1134\n",
       "Name: Height, dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#code to pull the specific error above\n",
    "df[\"Height\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Below is a key function that coverts bad height format and feet + inches into a standardized inches format. Bad ones will be given a 'NaN'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code to split the Height into feet and inches\n",
    "import re\n",
    "# this is a regex script which looks for a matching pattern in the input string. \n",
    "# https://docs.python.org/3/library/re.html\n",
    "r = re.compile(r\"([0-9]+)'([0-9]*\\.?[0-9]+|)\")\n",
    "def format_height(el):\n",
    "    el_new =el.replace(\" \",\"\")\n",
    "    m = r.match(el_new)\n",
    "    if m == None:\n",
    "        return float('NaN')\n",
    "    else:\n",
    "        return int(m.group(1))*12 + (0 if not m.group(2) else float(m.group(2)))\n",
    "    \n",
    "# Some of the records have been transcribed as mixed fractions rather than decimal values. These values have to be converted to inches using different Python formula as discussed below.    \n",
    "from fractions import Fraction\n",
    "def format_height_type2(el):  \n",
    "    el_new =el.replace('\"',\"\")\n",
    "    el_new =el_new.split('\\'')\n",
    "    el_new = [word for line in el_new for word in line.split()]\n",
    "    if not el_new:\n",
    "        return float('NaN')\n",
    "    else:\n",
    "        return int(el_new[0])*12 + (float(el_new[1]) + float(Fraction(el_new[2])))    \n",
    "# Below command starts with the beginning indentation indicating a new set of commands outside of the function, even if its in the same cell block like shown here.\n",
    "# The 'apply' function applies the function definted above to the data frame's each records' Prior Status field avlue. \n",
    "df[\"Height_Inches\"] = df[\"Height\"].astype(str).apply(lambda x: format_height_type2(x) if x.find('/') != -1 else format_height(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    22320.00000\n",
       "mean        64.36112\n",
       "std          4.13348\n",
       "min         21.50000\n",
       "25%         62.00000\n",
       "50%         64.50000\n",
       "75%         67.00000\n",
       "max        145.00000\n",
       "Name: Height_Inches, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show descriptive statistics\n",
    "df[\"Height_Inches\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 63.        ,  67.75      ,  57.5       ,  61.5       ,\n",
       "        66.5       ,  61.        ,  66.25      ,  66.        ,\n",
       "                nan,  70.        ,  62.        ,  63.25      ,\n",
       "        63.75      ,  64.5       ,  65.25      ,  65.75      ,\n",
       "        67.5       ,  67.        ,  68.        ,  68.5       ,\n",
       "        60.        ,  69.        ,  73.        ,  63.5       ,\n",
       "        61.25      ,  64.        ,  65.        ,  71.        ,\n",
       "        65.5       ,  64.75      ,  59.        ,  62.25      ,\n",
       "        62.75      ,  69.25      ,  58.        ,  59.25      ,\n",
       "        70.5       ,  62.5       ,  60.5       ,  64.25      ,\n",
       "        59.5       ,  61.75      ,  72.        ,  69.75      ,\n",
       "        59.75      ,  60.75      ,  60.125     ,  61.24      ,\n",
       "        67.125     ,  66.75      ,  67.25      ,  69.5       ,\n",
       "        68.75      ,  60.25      ,  61.375     ,  57.        ,\n",
       "        71.25      ,  56.        ,  58.25      ,  57.75      ,\n",
       "        68.25      ,  71.5       ,  66.6       ,  64.125     ,\n",
       "        64.625     ,  71.75      ,  65.875     ,  72.5       ,\n",
       "        70.75      ,  72.25      ,  56.5       ,  56.75      ,\n",
       "        57.25      ,  72.75      ,  73.25      ,  73.75      ,\n",
       "        72.375     ,  73.875     ,  24.        ,  50.        ,\n",
       "        52.        ,  60.24      ,  66.625     ,  58.75      ,\n",
       "        68.125     ,  47.75      ,  47.        ,  58.125     ,\n",
       "        58.5       ,  59.125     ,  60.3       ,  70.25      ,\n",
       "        73.5       ,  74.        ,  75.        ,  61.625     ,\n",
       "        61.2       ,  62.125     ,  55.75      ,  48.75      ,\n",
       "        64.875     ,  54.5       ,  48.        ,  46.5       ,\n",
       "        61.1       ,  54.25      ,  66.125     ,  53.        ,\n",
       "        54.        ,  55.        ,  48.5       ,  51.        ,\n",
       "        55.25      ,  55.5       ,  60.05      ,  44.125     ,\n",
       "        40.        ,  45.        ,  50.5       ,  51.75      ,\n",
       "        92.75      ,  62.3       ,  42.        ,  69.625     ,\n",
       "        60.625     ,  65.125     ,  63.125     ,  75.25      ,\n",
       "        81.4       ,  62.15      ,  63.625     ,  49.        ,\n",
       "        51.5       ,  68.27272727, 145.        ,  76.        ,\n",
       "        69.125     ,  68.875     ,  36.        ,  41.5       ,\n",
       "        66.4       ,  53.5       ,  53.75      ,  27.        ,\n",
       "        39.25      ,  64.73      ,  65.34      ,  58.625     ,\n",
       "        64.2       ,  61.125     ,  75.5       ,  66.875     ,\n",
       "        46.        ,  69.875     ,  74.25      ,  63.875     ,\n",
       "        64.74      ,  52.5       ,  52.25      ,  53.25      ,\n",
       "        54.75      ,  56.25      ,  57.625     ,  60.875     ,\n",
       "        65.625     ,  71.625     ,  62.375     ,  62.625     ,\n",
       "        33.5       ,  38.        ,  39.5       ,  44.5       ,\n",
       "        59.05      ,  66.375     ,  47.25      ,  59.625     ,\n",
       "        60.275     ,  60.375     ,  61.875     ,  74.5       ,\n",
       "        49.25      ,  77.        ,  30.5       ,  40.75      ,\n",
       "        31.        ,  62.675     ,  75.75      ,  68.33      ,\n",
       "        59.875     ,  68.625     ,  67.875     , 108.75      ,\n",
       "        66.2       ,  63.2       ,  69.2       ,  69.4       ,\n",
       "        78.5       ,  78.        ,  62.2       ,  35.        ,\n",
       "        41.        ,  43.        ,  44.        ,  26.        ,\n",
       "        67.675     ,  60.66      ,  21.5       ,  45.5       ,\n",
       "        49.75      ,  65.3       ,  37.5       ,  65.2       ,\n",
       "        72.05      ,  72.125     ,  70.125     ,  65.375     ,\n",
       "        63.375     ,  63.74      ,  34.5       ,  39.        ,\n",
       "        34.        ,  47.5       ,  33.        ,  36.5       ,\n",
       "        26.75      ,  32.        ,  28.5       ,  30.        ,\n",
       "        29.        ,  42.5       ,  37.        ,  29.5       ,\n",
       "        49.5       ,  40.5       ,  32.5       ,  31.5       ,\n",
       "        48.92      ,  38.5       ])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show distinct height values\n",
    "df[\"Height_Inches\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A study by (Margo & Steckel, 1982), which performed an analysis of the height vs age from the EnSlaved Mainfest data of around 50000+ enslaved people shipped between 1811 and 1861 to ports like Baltimore, Richmond and other cities from the Port of Savannah. According to this study, the average heights of tallest enslaved people was around 67 inches. In the same study where another set of Enslaved People's appraisal records showed the maximum height was found to be around 72 inches. Found below are the images from this study showing the different heights by age.\n",
    "\n",
    "![Height_by_age](Pics/Height_Enslaved_Manifests.png \"Height_Enslaved_Manifests\")\n",
    "\n",
    "![Max_Height_by_Weight](Pics/Height_Mississippi_chart.PNG \"Height_Mississippi_Chart\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above charts raise doubts on the unique values we observed to be higher than 80 inches and lesser than 5 inches. Thus, separating these records from the dataframe below shows the different representation of the Height during transcription."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DataItem</th>\n",
       "      <th>Height</th>\n",
       "      <th>Height_Inches</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5032</th>\n",
       "      <td>5034</td>\n",
       "      <td>4' 44.75\"</td>\n",
       "      <td>92.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5625</th>\n",
       "      <td>5627</td>\n",
       "      <td>5'2 1.4\"</td>\n",
       "      <td>81.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6197</th>\n",
       "      <td>6199</td>\n",
       "      <td>5'85.\"</td>\n",
       "      <td>145.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15694</th>\n",
       "      <td>15696</td>\n",
       "      <td>9'.75\"</td>\n",
       "      <td>108.75</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       DataItem     Height  Height_Inches\n",
       "5032       5034  4' 44.75\"          92.75\n",
       "5625       5627   5'2 1.4\"          81.40\n",
       "6197       6199     5'85.\"         145.00\n",
       "15694     15696     9'.75\"         108.75"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# code to show bad records\n",
    "df.loc[(df[\"Height_Inches\"]>80)|(df[\"Height_Inches\"]<5),['DataItem','Height','Height_Inches']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above values have to be manually handled by looking into the scanned documents and finding their right values as discussed below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of entries shown above where the height was mentioned as 4 feet 44.75 inches belonged to the enslaved person Milly Farmer c477-2, page 200, upon looking at the scanned document it was really captured as 4 feet 11.75 inches as found below from the document:\n",
    "\n",
    "![HeightIssue4](Pics/CoF_Data_Preparation_Height_Incorrect4.PNG \"CoF Height Error4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, it should be noted that there is record which was entered with a height of 9'.75\" which clearly seems like an impossible value. This had to be handled by attempting to manually look at the Certificate of Freedom record from the scanned documents. Upon analyzing we found that there was no CoF scanned document found for this person (Cof ID: 15696). It mentions that under note that this person was manumitted but we could not find the documents under Manumitted records as well. Hence, the height record was changed as NaN for this record."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DataItem</th>\n",
       "      <th>Height</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7859</th>\n",
       "      <td>7861</td>\n",
       "      <td>5 5\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11492</th>\n",
       "      <td>11494</td>\n",
       "      <td>illegible</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12837</th>\n",
       "      <td>12839</td>\n",
       "      <td>5\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15175</th>\n",
       "      <td>15177</td>\n",
       "      <td>5\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16964</th>\n",
       "      <td>16966</td>\n",
       "      <td>5\"1\"</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       DataItem     Height\n",
       "7859       7861       5 5\"\n",
       "11492     11494  illegible\n",
       "12837     12839         5\"\n",
       "15175     15177         5\"\n",
       "16964     16966       5\"1\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Code to show bad records \n",
    "df.loc[(df[\"Height_Inches\"].isna())&(df[\"Height\"].notna()),[\"DataItem\",\"Height\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other data capture issues were corrected by looking at the original scanned CoF as shown below: the height was noted as 5 5” which was in fact 5” 5’ - 5 feet 5 inches\n",
    "![HeightIssue2](Pics/CoF_Data_Preparation_Height_Incorrect2.PNG \"CoF Height Error2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From above code result, we identify that there are some invalid representations of the height where the transcribers did not follow the procedures to enter single quotes for Feet and double quotes for inches. These have to be manually handled as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code to manually update the issues identified above with the corrected value in inches\n",
    "# We directly use the dataitem id as shown above to update the records.\n",
    "df[\"Height_Inches\"]\n",
    "df.loc[(df[\"DataItem\"]==5034), \"Height_Inches\"] =59.75\n",
    "df.loc[(df[\"DataItem\"]==5627), \"Height_Inches\"] =63.40\n",
    "df.loc[(df[\"DataItem\"]==6199), \"Height_Inches\"] =None\n",
    "df.loc[(df[\"DataItem\"]==15696), \"Height_Inches\"] =None\n",
    "df.loc[(df[\"DataItem\"]==7861), \"Height_Inches\"] =65.00\n",
    "df.loc[(df[\"DataItem\"]==11494), \"Height_Inches\"] =None\n",
    "df.loc[(df[\"DataItem\"]==12839), \"Height_Inches\"] =60.00\n",
    "df.loc[(df[\"DataItem\"]==15177), \"Height_Inches\"] =60.00\n",
    "df.loc[(df[\"DataItem\"]==16966), \"Height_Inches\"] =61.00"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DataItem</th>\n",
       "      <th>Height</th>\n",
       "      <th>Height_Inches</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6197</th>\n",
       "      <td>6199</td>\n",
       "      <td>5'85.\"</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11492</th>\n",
       "      <td>11494</td>\n",
       "      <td>illegible</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15694</th>\n",
       "      <td>15696</td>\n",
       "      <td>9'.75\"</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       DataItem     Height  Height_Inches\n",
       "6197       6199     5'85.\"            NaN\n",
       "11492     11494  illegible            NaN\n",
       "15694     15696     9'.75\"            NaN"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show height values that are outliers and invalid values\n",
    "df.loc[(df[\"Height_Inches\"]>80)|(df[\"Height_Inches\"]<5),['DataItem','Height','Height_Inches']]\n",
    "df.loc[(df[\"Height_Inches\"].isna())&(df[\"Height\"].notna()),[\"DataItem\",\"Height\",\"Height_Inches\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As could be seen, wherever possible, the team went through the scanned documents for the erroneous entries and updated the conversions ourselves. For these erroneous data, MSA would be contacted to fix the Source of Record."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Age feature \n",
    "Age field was originally in the text type format, converted to number, and converted all the decimals which was entered as it is from the original document listed as months into a 12 month per year relative decimal value, for example, the original CoF noted the enslaved person as 18 months old, the dataset had this value as 0.18 under the age column which actually should be 1.5 years old."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Below is a key function that converts bad decimal values transcrbed into a good decimal fraction of a year. For example, 0.18 was transcribed for 18 months, this was converted to 1.5 years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# code to pull the above \n",
    "import math\n",
    "from fractions import Fraction\n",
    "def format_fraction_age(x):  \n",
    "    fractional, whole = math.modf(x)\n",
    "    # Some of the records have been transcribed as bad fractions, this will fix it as a good fraction of a year\n",
    "    return float(fractional*100/12)   \n",
    "# The 'apply' function applies the function definted above to the data frame's each records' Age field value. \n",
    "df[\"AgeFormatted\"] = df[\"Age\"].apply(lambda x: float(x) if x >= 1 else format_fraction_age(x))\n",
    "# import shlex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DataItem</th>\n",
       "      <th>Age</th>\n",
       "      <th>AgeFormatted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11876</th>\n",
       "      <td>11878</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13870</th>\n",
       "      <td>13872</td>\n",
       "      <td>0.18</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19625</th>\n",
       "      <td>19627</td>\n",
       "      <td>0.80</td>\n",
       "      <td>6.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22819</th>\n",
       "      <td>22822</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22849</th>\n",
       "      <td>22852</td>\n",
       "      <td>0.18</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22861</th>\n",
       "      <td>22864</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22890</th>\n",
       "      <td>22893</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22907</th>\n",
       "      <td>22910</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22912</th>\n",
       "      <td>22915</td>\n",
       "      <td>0.18</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22940</th>\n",
       "      <td>22943</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22969</th>\n",
       "      <td>22972</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.583333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22979</th>\n",
       "      <td>22982</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23038</th>\n",
       "      <td>23041</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.416667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23059</th>\n",
       "      <td>23062</td>\n",
       "      <td>0.18</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23118</th>\n",
       "      <td>23121</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23126</th>\n",
       "      <td>23129</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.583333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23154</th>\n",
       "      <td>23157</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23171</th>\n",
       "      <td>23174</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23269</th>\n",
       "      <td>23272</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       DataItem   Age  AgeFormatted\n",
       "11876     11878  0.08      0.666667\n",
       "13870     13872  0.18      1.500000\n",
       "19625     19627  0.80      6.666667\n",
       "22819     22822  0.09      0.750000\n",
       "22849     22852  0.18      1.500000\n",
       "22861     22864  0.03      0.250000\n",
       "22890     22893  0.08      0.666667\n",
       "22907     22910  0.03      0.250000\n",
       "22912     22915  0.18      1.500000\n",
       "22940     22943  0.06      0.500000\n",
       "22969     22972  0.07      0.583333\n",
       "22979     22982  0.06      0.500000\n",
       "23038     23041  0.05      0.416667\n",
       "23059     23062  0.18      1.500000\n",
       "23118     23121  0.03      0.250000\n",
       "23126     23129  0.07      0.583333\n",
       "23154     23157  0.06      0.500000\n",
       "23171     23174  0.04      0.333333\n",
       "23269     23272  0.06      0.500000"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# code to find the converted Age records\n",
    "df.loc[(df[\"Age\"]<1),['DataItem','Age','AgeFormatted']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For one case which was listed to be as 100 years old, upon checking the CoF original document, it's unclear as the document shows something like eighty & twenty years as highlighted below: This is also noted in the notes section as “Age given as eighty and twenty years. Could potentially be 28 years, not 100.”\n",
    "\n",
    "![AgeIssue](Pics/CoF_Data_Preparation_Age_Incorrect.PNG \"CoF Age Error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the output file\n",
    "# save the output to the csv\n",
    "dfo = pd.DataFrame(df)\n",
    "dfo.to_csv('Datasets/LoS_Clean_Output_Mod2.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebooks\n",
    "\n",
    "The below module is organized into a sequential set of Python Notebooks that allows to interact with the Legacy of Slavery's Certificates of Freedom collection by exploring, cleaning, preparing, visualizing and analysing it from historical context perspective. \n",
    "\n",
    "3. [Certificates Of Freedom: Context Based Data Visualization](LoS_CoF_Module3.ipynb) **(If you get a 500 Internal Server Error, please refresh the browser to render this module again)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Click here to go the Next Module](LoS_CoF_Module3.ipynb) **(If you get a 500 Internal Server Error, please refresh the browser to render this module again)**\n",
    "\n",
    "[Click here to go the Previous Module](LoS_CoF_Module1.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}