1.3_lesson3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ray's Python Class Lesson 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm glad you've made it so far, we're almost half way done through the basic issues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By the end of this notebook, you can use python to do very practical analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let me officially introduce you 2 crutial library to the data science:\n",
    "\n",
    "### pandas and numpy\n",
    "\n",
    "9 out of 10, people import them like this:\n",
    "\n",
    "#### import pandas as pd\n",
    "\n",
    "#### import numpy as np\n",
    "\n",
    "Or you can rename it as you like:\n",
    "\n",
    "import pandas as itsaHugePandaButTheNameIsReallyLongSoWeRarelyDoItThisWay\n",
    "\n",
    "Now with pandas and numpy, You can import data files and do hefty calculations on them, I don't have to give you toy data any more\n",
    "\n",
    "=================================\n",
    "\n",
    "Beware, pandas and numpy are not python built-in library, you have to install them. The way installing most python library is... not that complicated. In linxu and mac os:\n",
    "\n",
    "$pip install pandas\n",
    "\n",
    "$pip install numpy\n",
    "\n",
    "If you are in python3,use pip3.\n",
    "\n",
    "If you're using Anaconda, it's already installed\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, how to open an excel file or csv file?\n",
    "\n",
    "If the file size isn't beyond what your computer memory can handle, it's simple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For Excel file\n",
    "exldf=pd.read_excel(\"/path/to/your/file.xlsx\")\n",
    "#### For Csv File\n",
    "csvdf=pd.read_csv(\"../relative/path/to/your/file.csv\")\n",
    "\n",
    "A little catching up on file path on linux system:\n",
    "\n",
    "if you have a file a.xlsx under the same folder as python code, path is \"a.xlsx\"\n",
    "\n",
    "if your python file is under folder \"pythoncodes\", and your excel sheet a.xlsx is under folder \"excels\", these 2 folders are under the same root folder \"/data\", you can either use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ready for the real shit?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now in the same folder there is a csv file about [US Permanent Visa Applications](https://www.kaggle.com/jboysen/us-perm-visas), file name is \"us_perm_visas.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vs_df=pd.read_csv(\"us_perm_visas.csv\") # The file is kinda big, load it for about 10 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ignore error message for now. If you are reading excel format, use the similar syntex read_excel\n",
    "\n",
    "Variable \"vs_df\" here is a pandas dataframe, pandas dataframe is, in plain English, something like a structured excel sheet.\n",
    "\n",
    "Actually [pandas is versatile and capable of reading many formats](https://pandas.pydata.org/pandas-docs/stable/api.html#input-output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read first 10 rows of the df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vs_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do you check the last 7 lines of a dataframe?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vs_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can perform many operations in very limited lines of codes, providing you know the codes, it's a good thing if you mark down the pandas documentation site"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series\n",
    "\n",
    "Series is like a column of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vs_df[\"us_economic_sector\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the distribution of sectors, let's say, the top values of us_economic_sector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vs_df[\"us_economic_sector\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, all the more reason to learn python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are not familiar with the idea series, don't be afraid, you can always retreat your comfort zone by turning it into a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sectors=list(vs_df[\"us_economic_sector\"])\n",
    "print(sectors[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the distribution of column \"wage_offer_unit_of_pay_9089\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose you want to compare between the offers by column \"wage_offer_from_9089\", but all these different time unites makes this effort futile, some yearly offer is around 100000, but some hourly rate is 10.\n",
    "\n",
    "That's when pandas comes in handy in data processing.\n",
    "\n",
    "First,we list out the time unit in a dictionary, about how many days a time unit represent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unit2day=dict({\n",
    "    \"Year\":250, # 250 working days, not calendar 365 days a year\n",
    "    \"yr\":250,\n",
    "    \"Hour\":0.125, # assuming 8 working hours a day\n",
    "    \"hr\":0.125,\n",
    "    \"Week\":5,\n",
    "    \"Month\":22.5,\n",
    "    \"wk\":5,\n",
    "    \"mth\":22.5,\n",
    "    \"Bi-Weekly\":10,\n",
    "    \"bi\":10\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, before I do this, I should officially introduce you the function, and we'll return to this task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though you may know the simple syntax of a python function, I have to introduce you the majestic idea of function, the basic idea is about wraping thing up, and don't teach the programe twice to do the same task.\n",
    "\n",
    "Imaging computer is child, you spend 20 minutes telling him how to carry out the garbage, you can wrap it up in a function, so next time, you tell that boy to carry out the garbage, you apply the function, with no need to that 20 minutes explaining.\n",
    "\n",
    "in python you do,def is stort for define:\n",
    "\n",
    "def throwItToTheGarbageDump():\n",
    "    \n",
    "    python code here xxxxxxxxxx\n",
    "    \n",
    "    python code here xxxxxxxxxx\n",
    "    \n",
    "    print(\"Throwed out\") #notice the indent of 4 spaces\n",
    "    \n",
    "    return xxxx\n",
    "    \n",
    "every time you use:\n",
    "\n",
    "throwItToTheGarbageDump()\n",
    "\n",
    "The boy will carry the garbage\n",
    "\n",
    "#### Parameter\n",
    "\n",
    "say you want him to throw out a huge box of newspaper, the shape and content is different, but the task is pretty much the same, so you change the function a little bit:\n",
    "\n",
    "def throwItToTheGarbageDump(stuff):\n",
    "    \n",
    "    Python code here\n",
    "    \n",
    "    remove stuff here\n",
    "    .....\n",
    "\n",
    "Then you use throwItToTheGarbageDump(\"newspaper\"), job done\n",
    "\n",
    "#### Default parameter\n",
    "\n",
    "Of course you don't have to tell him what stuff to throw out everything, if not specified, you throw out the garbage, when you do specify, he will comply accordingly\n",
    "\n",
    "def throwItToTheGarbageDump(stuff=\"garbage\"):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define something simple in one line\n",
    "def find_square(x):return x**2\n",
    "# use the function\n",
    "find_square(25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# or we do it in multiple rows\n",
    "# something more complicated\n",
    "# get if and else condition considered\n",
    "def factorial(n):\n",
    "    n=int(n)\n",
    "    if n<1:\n",
    "        return \"The number doesn't have factorial\"\n",
    "    else:\n",
    "        answer=1\n",
    "        for i in list(range(1,n+1)):\n",
    "            answer=i*answer\n",
    "        return answer\n",
    "    \n",
    "factorial(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above function return a factorial of a nature number, what's the biggest problem you see in this function?\n",
    "\n",
    "The biggest problem is, somebody sure has written the factorial function, we can just use that.\n",
    "\n",
    "The point is, most time you see () in python code, is we use a function, defined either by our codes or some other library , or a function in a class. () is the way to deploy that function.\n",
    "\n",
    "In this case, if you google for a minute, of course you'll find the factorial function, it's in the math library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from math import factorial\n",
    "\n",
    "factorial(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, I won't give you the \"don't reinvent wheel\" speech, but using other people's code will save you some quality time, unless you do it for excercise purpose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rename the function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the function name is too long or too dumb for you, you can do it this way.\n",
    "\n",
    "The function won't activate its operation if you don't call it by (). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fc=factorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now you use the () to call the function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fc(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "do you know list is a built-in function, so just another funciton?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# so, we can do\n",
    "l=list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(l([1,2,3,4]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Map and lambda function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some cases you have to define a short function, to run it on a list , but you don't want to go too verbose on the task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say we have these unique values of sectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# remove the duplicate in the list sector\n",
    "\n",
    "# in jupyter you place the phrase %time ahead of code to measure the time comsumption of your operation.\n",
    "\n",
    "# .fillna() is how we fill the nan values in the series\n",
    "\n",
    "# notice how we chain the functions together, a series after .fillna() is still a series, and a series can do .unique()\n",
    "\n",
    "%time sectors_unique=vs_df[\"us_economic_sector\"].fillna(\"Nan\").unique()\n",
    "print(sectors)\n",
    "\n",
    "# If you know you want to use fillna, but you don't know how to use it.\n",
    "# Here's a jupyter hot key comes in handy! You can click on fillna,\n",
    "# Then press \"shift + tab +tab +tab\", tab for three times\n",
    "# In jupyter shift tab for information, more tab for more information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you remember correctly,you might imagine you can get the unique values by the following way of using set.\n",
    "\n",
    "list(set(vs_df[\"us_economic_sector\"]))\n",
    "\n",
    "Well, don't. The csv file you opened is 299mb in size, 374,362 rows of data, that set() calculation could let my server burn for a while. But the .unique() function finished in an instant.\n",
    "\n",
    "Because python iteration/calculation might be fast to type, but it's slow to run, it works well if the data is small, but when data is huge, you can process data with pandas dataframe's built-in functions, because they will toss the heavy job to c/c++, will run much faster.\n",
    "\n",
    "Most data scientist favour python over other languages, because it let you focus on the task logic and your own complication on business. In years smart people developed excellent tools, leting this language run big tasks as fast as c/c++. Pandas, numpy or even machine learning libraries like sklearn, tensorflow, theano, pytorch, they all do the hefty calculations in c/c++."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say we have these unique values of sectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "list(sectors_unique)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to take the first 3 letters of each word, I believe you are capable enough to write this simple function yourself, you can complete the function here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def take3letters(word):\n",
    "    \"\"\"\n",
    "    This is the comment/menu in the function\n",
    "    Usually dwelled between 3 pairs of quotes,\n",
    "    At the start of a function\n",
    "    \n",
    "    You can write here what you want to tell the user about your function.\n",
    "    What kind of/How many parameters/args they want to put in\n",
    "    What kind of return they expect to see.\n",
    "    \"\"\"\n",
    "    return "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "well, having fun? Now we can apply your function on map\n",
    "\n",
    "map can apply your specific function on each element of the list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sectors_short=list(map(take3letters,sectors_unique))\n",
    "print(sectors_short)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's simple enough to use def to define a funtion\n",
    "\n",
    "No trouble with public/private\n",
    "\n",
    "Don't have to declare variable type\n",
    "\n",
    "But it can be even simpler\n",
    "\n",
    "Use the lambda function, you write simple functions in oneline, and you don't have to use the word return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sectors_short=list(map(lambda x:x if len(x)<4 else x[:3],sectors_unique))\n",
    "print(sectors_short)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"lambda x:x if len(x) < 4 else x[:3] \" is a function, uncalled\n",
    "\n",
    "Knowing that much about function, let's get back on the problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The column tells about the salary/ wage, check the first 12 rows\n",
    "vs_df[\"wage_offer_from_9089\"].head(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use .apply() to apply your function to every value of the series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# x!=0 means x not equals to 0\n",
    "# create a new column to fill in the \"time unit to day\" value in the dictionary ”unit2day“ we created\n",
    "vs_df[\"unit_2_day\"]=vs_df[\"wage_offer_unit_of_pay_9089\"].fillna(1).apply(lambda x: unit2day[x] if x!=1 else 1)\n",
    "vs_df[\"unit_2_day\"].head(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "World isn't perfect, in data science, it's more so.\n",
    "\n",
    "Check closely you can find many data in column \"wage_offer_from_9098\" isn't even number.\n",
    "\n",
    "We have to turn it into clean float data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def turntofloat(x):\n",
    "    \"\"\"\n",
    "    Turn dirty data formats (int,str in this case)\n",
    "    Into float for the sake of calculation\n",
    "    args[0] an input you want to transfer\n",
    "    \"\"\"\n",
    "    try:\n",
    "        return float(x) # return the float format of x\n",
    "    except:\n",
    "        return float(0) # if any boogy happens under try:, we run the backup plan in except:\n",
    "vs_df[\"wage_offer_from_9089\"]=vs_df[\"wage_offer_from_9089\"].apply(turntofloat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# float() is to make sure the calculation is between float and float,\n",
    "vs_df[\"offer_price\"]=vs_df.apply(lambda row: float(row[\"wage_offer_from_9089\"])/float(row[\"unit_2_day\"]) if row[\"unit_2_day\"]!=0 else 0, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check result, you can see offer_price is a salary number in daily sense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vs_df[[\"offer_price\",\"wage_offer_from_9089\",\"wage_offer_unit_of_pay_9089\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can say , you can do that with spreadsheet, probably using a vlookup combined with other formula.\n",
    "\n",
    "But keep practice your pandas skill, the power shall transcend excel quickly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Speaking of excel, you can do [pivot table in pandas](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.pivot_table.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the average offer_price to compare between sectors, since you already got the offer price in a uniformed time unit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vs_cross=pd.pivot_table(vs_df[[\"us_economic_sector\",\"offer_price\"]],index=\"us_economic_sector\",dropna=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vs_cross=vs_cross.sort_values(by=\"offer_price\",ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# now save the result of your effort\n",
    "vs_cross.to_excel(\"salary_by_sector.xlsx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check your folder to see if the file is there, you can download the file with jupyter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Use the column \"case_status\" to calculate the accept rate of each sector, export your result to \"ac_rate_by_sector.xlsx\"\n",
    "2. Use any column of data you like to choose, to show what kind of people got accepted more often. Show your codings in this notebook\n",
    "\n",
    "\n",
    "#### Extra point (out of the lesson today, but to be a great analyst / data scientist, please spend time pondering how)\n",
    "3. Use any statistic knowledge you know to predict a given person's fate of acceptence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}