7. Data Aggregation and Group Operations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "#### Categorizing a dataset and applying a function to each group (aggregation or transformation) is a critical component os data analysis workflow.\n",
    "#### After 'loading', 'merging' and 'preparing' a dataset, you may need to compute 'group statistics' or 'pivot' tables for reporting or visualization purposes.\n",
    "#### A reason why SQL is so popular is because of the ease with which data can be joined, filtered, transformed and agregated. But they are somewhat contrained in group operations.\n",
    "#### With the expresiveness of Python and pandas we can perform complex group operations by using functions that accept pandas object or NumPy array.\n",
    "#### NOTE - Aggregation of time series data is special case of 'groupby', also refered to as 'resampling'. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GroupBy Mechanics\n",
    "#### The 'split-apply-combine' term describes group operations succinctly.\n",
    "#### In the first stage, data in pandas object is 'split' into groups based on one or more keys. Splitting is performed on particular axis of the object.\n",
    "#### Next, a function is 'applied' to each group producing a new value.\n",
    "#### Finally, the results of those applications are 'combined' into a result object. The form of the result will depend on what is being done to the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Grouping key can take many forms. Keys can be of different types:\n",
    "####     1. A list of values having same length as the axis being grouped.\n",
    "####     2. A value indicating column name in a DataFrame.\n",
    "####     3. A dict or Series providing correspondence between values on the axis being grouped an group names.\n",
    "####     4. A function to be invoked on axis index or individual labels in the index.\n",
    "#### Note that last three methods are actually shortcuts for producing array of values to split up the object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "      <th>data1</th>\n",
       "      <th>data2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>one</td>\n",
       "      <td>1.033605</td>\n",
       "      <td>-0.208085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a</td>\n",
       "      <td>two</td>\n",
       "      <td>-0.414086</td>\n",
       "      <td>0.026467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b</td>\n",
       "      <td>one</td>\n",
       "      <td>-0.590407</td>\n",
       "      <td>0.812162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b</td>\n",
       "      <td>two</td>\n",
       "      <td>0.057372</td>\n",
       "      <td>0.365072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a</td>\n",
       "      <td>one</td>\n",
       "      <td>-0.511034</td>\n",
       "      <td>2.224576</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key1 key2     data1     data2\n",
       "0    a  one  1.033605 -0.208085\n",
       "1    a  two -0.414086  0.026467\n",
       "2    b  one -0.590407  0.812162\n",
       "3    b  two  0.057372  0.365072\n",
       "4    a  one -0.511034  2.224576"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.DataFrame({'key1': ['a', 'a', 'b', 'b', 'a'],\n",
    "                  'key2': ['one', 'two', 'one', 'two', 'one'],\n",
    "                  'data1': np.random.randn(5),\n",
    "                  'data2': np.random.randn(5)})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Computing mean of the data1 column of above DataFrame object using labels from key2 can be done in a number of ways.\n",
    "#### One way is to access data1 and call 'groupby' with the column (Series) at key1. This will return a GroupBy object.\n",
    "#### The object has not computed anything except for intermediate data about the group key 'key1'. It has all the information to apply some operation to each of the groups. eg - 'mean' method.\n",
    "#### Important thing to note is that the Series has been aggregated by a group key, producing a new Series indexed by unique values in key1 column. Hence the result index also has the name 'key1'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.groupby.groupby.SeriesGroupBy object at 0x000001DF97743278>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped = df['data1'].groupby(df['key1'])\n",
    "grouped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key1\n",
       "a    0.036162\n",
       "b   -0.266518\n",
       "Name: data1, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If we had passed multiple arrays in the groupby column as list, we get a resulting Series with hierarchical index.\n",
    "#### This index consists of unique arrangement of keys passed in the groupby method.\n",
    "#### The group keys can be any arrays of the right length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key1  key2\n",
       "a     one     0.261286\n",
       "      two    -0.414086\n",
       "b     one    -0.590407\n",
       "      two     0.057372\n",
       "Name: data1, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "means = df['data1'].groupby([df['key1'], df['key2']]).mean()\n",
    "means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>key2</th>\n",
       "      <th>one</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>key1</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0.261286</td>\n",
       "      <td>-0.414086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>-0.590407</td>\n",
       "      <td>0.057372</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "key2       one       two\n",
       "key1                    \n",
       "a     0.261286 -0.414086\n",
       "b    -0.590407  0.057372"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "means.unstack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "California  2005   -0.414086\n",
       "            2006   -0.590407\n",
       "Ohio        2005    0.545488\n",
       "            2006   -0.511034\n",
       "Name: data1, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states = np.array(['Ohio', 'California', 'California', 'Ohio', 'Ohio'])\n",
    "years = np.array([2005, 2005, 2006, 2005, 2006])\n",
    "df['data1'].groupby([states, years]).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mostly the grouping information would be present in the same data you want to work on. In this case, you can pass column names as group keys.\n",
    "#### If you have a non numeric data in a column, it is called a 'nuisance column' and it is excluded from the aggregation result. But we can use these columns for grouping.\n",
    "#### By default, all numeric columns are aggregated. Although it is possible to filter down to a subset.\n",
    "#### Regardless of objective, a useful group method is 'size', which return a Series having group sizes. Missing values in a group key are excluded from the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>data1</th>\n",
       "      <th>data2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>key1</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0.036162</td>\n",
       "      <td>0.680986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>-0.266518</td>\n",
       "      <td>0.588617</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         data1     data2\n",
       "key1                    \n",
       "a     0.036162  0.680986\n",
       "b    -0.266518  0.588617"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('key1').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>data1</th>\n",
       "      <th>data2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">a</th>\n",
       "      <th>one</th>\n",
       "      <td>0.261286</td>\n",
       "      <td>1.008245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>-0.414086</td>\n",
       "      <td>0.026467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">b</th>\n",
       "      <th>one</th>\n",
       "      <td>-0.590407</td>\n",
       "      <td>0.812162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>0.057372</td>\n",
       "      <td>0.365072</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              data1     data2\n",
       "key1 key2                    \n",
       "a    one   0.261286  1.008245\n",
       "     two  -0.414086  0.026467\n",
       "b    one  -0.590407  0.812162\n",
       "     two   0.057372  0.365072"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(['key1', 'key2']).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key1  key2\n",
       "a     one     2\n",
       "      two     1\n",
       "b     one     1\n",
       "      two     1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(['key1', 'key2']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iterating Over Groups\n",
    "#### GroupBy objects support iteration, generating sequence of 2-tuples having group name with chunk of data.\n",
    "#### In case of multiple keys, first element in the tuple will be tuple of keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a\n",
      "  key1 key2     data1     data2\n",
      "0    a  one  1.033605 -0.208085\n",
      "1    a  two -0.414086  0.026467\n",
      "4    a  one -0.511034  2.224576\n",
      "b\n",
      "  key1 key2     data1     data2\n",
      "2    b  one -0.590407  0.812162\n",
      "3    b  two  0.057372  0.365072\n"
     ]
    }
   ],
   "source": [
    "for name, group in df.groupby('key1'):\n",
    "    print(name)\n",
    "    print(group)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('a', 'one')\n",
      "  key1 key2     data1     data2\n",
      "0    a  one  1.033605 -0.208085\n",
      "4    a  one -0.511034  2.224576\n",
      "('a', 'two')\n",
      "  key1 key2     data1     data2\n",
      "1    a  two -0.414086  0.026467\n",
      "('b', 'one')\n",
      "  key1 key2     data1     data2\n",
      "2    b  one -0.590407  0.812162\n",
      "('b', 'two')\n",
      "  key1 key2     data1     data2\n",
      "3    b  two  0.057372  0.365072\n"
     ]
    }
   ],
   "source": [
    "for (k1, k2), group in df.groupby(['key1', 'key2']):\n",
    "    print((k1, k2))\n",
    "    print(group)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can do whatever we want with the chunks of data. A useful thing would be to compute a dict of the data chunks as a one-liner.\n",
    "#### By default 'groupby' groups on axis=0, but you can group on any other axes. \n",
    "#### We can print out the groups too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a':   key1 key2     data1     data2\n",
       " 0    a  one  1.033605 -0.208085\n",
       " 1    a  two -0.414086  0.026467\n",
       " 4    a  one -0.511034  2.224576, 'b':   key1 key2     data1     data2\n",
       " 2    b  one -0.590407  0.812162\n",
       " 3    b  two  0.057372  0.365072}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pieces = dict(list(df.groupby('key1')))\n",
    "pieces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "      <th>data1</th>\n",
       "      <th>data2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b</td>\n",
       "      <td>one</td>\n",
       "      <td>-0.590407</td>\n",
       "      <td>0.812162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b</td>\n",
       "      <td>two</td>\n",
       "      <td>0.057372</td>\n",
       "      <td>0.365072</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key1 key2     data1     data2\n",
       "2    b  one -0.590407  0.812162\n",
       "3    b  two  0.057372  0.365072"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pieces['b']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key1      object\n",
       "key2      object\n",
       "data1    float64\n",
       "data2    float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "float64\n",
      "      data1     data2\n",
      "0  1.033605 -0.208085\n",
      "1 -0.414086  0.026467\n",
      "2 -0.590407  0.812162\n",
      "3  0.057372  0.365072\n",
      "4 -0.511034  2.224576\n",
      "object\n",
      "  key1 key2\n",
      "0    a  one\n",
      "1    a  two\n",
      "2    b  one\n",
      "3    b  two\n",
      "4    a  one\n"
     ]
    }
   ],
   "source": [
    "grouped = df.groupby(df.dtypes, axis=1)\n",
    "\n",
    "for dtype, group in grouped:\n",
    "    print(dtype)\n",
    "    print(group)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting a Column or Subset of Columns\n",
    "#### Indexing a GroupBy object created from a DataFrame with column name or array of column names has same effect as column subsetting for aggregation.\n",
    "#### What it means is that doing this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\ndf.groupby('key1')['data1']\\ndf.groupby('key1')[['data2']]\\n\""
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "df.groupby('key1')['data1']\n",
    "df.groupby('key1')[['data2']]\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### is the syntactic sugar (i.e. has the same effect) for:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\ndf['data1'].groupby(df['key1'])\\ndf[['data2']].groupby(df['key1'])\\n\""
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "df['data1'].groupby(df['key1'])\n",
    "df[['data2']].groupby(df['key1'])\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For large datasets, it is desirable if we aggregate using only a few columns.\n",
    "#### The object returned by this indexing is a grouped DataFrame if a list or array is passed or a grouped Series if a single column is passed as scalar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>data2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">a</th>\n",
       "      <th>one</th>\n",
       "      <td>1.008245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>0.026467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">b</th>\n",
       "      <th>one</th>\n",
       "      <td>0.812162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>0.365072</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              data2\n",
       "key1 key2          \n",
       "a    one   1.008245\n",
       "     two   0.026467\n",
       "b    one   0.812162\n",
       "     two   0.365072"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(['key1', 'key2'])[['data2']].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.groupby.groupby.SeriesGroupBy object at 0x000001DF97A45710>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s_grouped = df.groupby(['key1', 'key2'])['data2']\n",
    "s_grouped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key1  key2\n",
       "a     one     1.008245\n",
       "      two     0.026467\n",
       "b     one     0.812162\n",
       "      two     0.365072\n",
       "Name: data2, dtype: float64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s_grouped.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouping with Dicts and Series\n",
    "#### Grouping information may be present in forms other than an array. Instead os passing an array, we can also pass a dict to groupby. Using unused grouping keys is OK.\n",
    "#### We can also use Series in groupby, which can be viewed as fixed-sized mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Joe</th>\n",
       "      <td>0.446110</td>\n",
       "      <td>-0.242829</td>\n",
       "      <td>-0.625072</td>\n",
       "      <td>-1.009086</td>\n",
       "      <td>-0.439756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>steve</th>\n",
       "      <td>0.138213</td>\n",
       "      <td>0.433581</td>\n",
       "      <td>-0.246506</td>\n",
       "      <td>0.017845</td>\n",
       "      <td>-0.937134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wes</th>\n",
       "      <td>1.832978</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.340975</td>\n",
       "      <td>-2.868593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jim</th>\n",
       "      <td>-1.802749</td>\n",
       "      <td>0.348062</td>\n",
       "      <td>-1.539908</td>\n",
       "      <td>0.668386</td>\n",
       "      <td>0.346682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Travis</th>\n",
       "      <td>0.140066</td>\n",
       "      <td>-1.241122</td>\n",
       "      <td>-2.421142</td>\n",
       "      <td>0.286051</td>\n",
       "      <td>0.797616</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               a         b         c         d         e\n",
       "Joe     0.446110 -0.242829 -0.625072 -1.009086 -0.439756\n",
       "steve   0.138213  0.433581 -0.246506  0.017845 -0.937134\n",
       "Wes     1.832978       NaN       NaN  0.340975 -2.868593\n",
       "Jim    -1.802749  0.348062 -1.539908  0.668386  0.346682\n",
       "Travis  0.140066 -1.241122 -2.421142  0.286051  0.797616"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "people = pd.DataFrame(np.random.randn(5,5),\n",
    "                     columns = ['a', 'b', 'c', 'd', 'e'],\n",
    "                     index = ['Joe', 'steve', 'Wes', 'Jim', 'Travis'])\n",
    "people.iloc[2:3, [1,2]] = np.nan # a few NA values\n",
    "\n",
    "people"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapping = {'a': 'red', 'b': 'red', 'c': 'blue',\n",
    "          'd': 'blue', 'e': 'red', 'f': 'orange'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>blue</th>\n",
       "      <th>red</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Joe</th>\n",
       "      <td>-1.634158</td>\n",
       "      <td>-0.236475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>steve</th>\n",
       "      <td>-0.228661</td>\n",
       "      <td>-0.365340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wes</th>\n",
       "      <td>0.340975</td>\n",
       "      <td>-1.035615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jim</th>\n",
       "      <td>-0.871522</td>\n",
       "      <td>-1.108005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Travis</th>\n",
       "      <td>-2.135091</td>\n",
       "      <td>-0.303440</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            blue       red\n",
       "Joe    -1.634158 -0.236475\n",
       "steve  -0.228661 -0.365340\n",
       "Wes     0.340975 -1.035615\n",
       "Jim    -0.871522 -1.108005\n",
       "Travis -2.135091 -0.303440"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "by_column = people.groupby(mapping, axis=1)\n",
    "by_column.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a       red\n",
       "b       red\n",
       "c      blue\n",
       "d      blue\n",
       "e       red\n",
       "f    orange\n",
       "dtype: object"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "map_series = pd.Series(mapping)\n",
    "\n",
    "map_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>blue</th>\n",
       "      <th>red</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Joe</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>steve</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wes</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jim</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Travis</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        blue  red\n",
       "Joe        2    3\n",
       "steve      2    3\n",
       "Wes        1    2\n",
       "Jim        2    3\n",
       "Travis     2    3"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "people.groupby(map_series, axis=1).count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouping with functions\n",
    "#### Functions is a more generic way of defining a group mapping. Every function passed as group key will be called once per index value, with th return values used as group names.\n",
    "#### Mixing functions with arrays, dicts or Series is not a problem as everything gets converted to arrays internally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.476339</td>\n",
       "      <td>0.105232</td>\n",
       "      <td>-2.164979</td>\n",
       "      <td>0.000274</td>\n",
       "      <td>-2.961667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.138213</td>\n",
       "      <td>0.433581</td>\n",
       "      <td>-0.246506</td>\n",
       "      <td>0.017845</td>\n",
       "      <td>-0.937134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.140066</td>\n",
       "      <td>-1.241122</td>\n",
       "      <td>-2.421142</td>\n",
       "      <td>0.286051</td>\n",
       "      <td>0.797616</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          a         b         c         d         e\n",
       "3  0.476339  0.105232 -2.164979  0.000274 -2.961667\n",
       "5  0.138213  0.433581 -0.246506  0.017845 -0.937134\n",
       "6  0.140066 -1.241122 -2.421142  0.286051  0.797616"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "people.groupby(len).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>one</th>\n",
       "      <td>0.446110</td>\n",
       "      <td>-0.242829</td>\n",
       "      <td>-0.625072</td>\n",
       "      <td>-1.009086</td>\n",
       "      <td>-2.868593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>-1.802749</td>\n",
       "      <td>0.348062</td>\n",
       "      <td>-1.539908</td>\n",
       "      <td>0.668386</td>\n",
       "      <td>0.346682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>one</th>\n",
       "      <td>0.138213</td>\n",
       "      <td>0.433581</td>\n",
       "      <td>-0.246506</td>\n",
       "      <td>0.017845</td>\n",
       "      <td>-0.937134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <th>two</th>\n",
       "      <td>0.140066</td>\n",
       "      <td>-1.241122</td>\n",
       "      <td>-2.421142</td>\n",
       "      <td>0.286051</td>\n",
       "      <td>0.797616</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              a         b         c         d         e\n",
       "3 one  0.446110 -0.242829 -0.625072 -1.009086 -2.868593\n",
       "  two -1.802749  0.348062 -1.539908  0.668386  0.346682\n",
       "5 one  0.138213  0.433581 -0.246506  0.017845 -0.937134\n",
       "6 two  0.140066 -1.241122 -2.421142  0.286051  0.797616"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key_list = ['one', 'one', 'one', 'two', 'two']\n",
    "\n",
    "people.groupby([len, key_list]).min()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouping by Index Levels\n",
    "#### Another convinience for hierarchically indexed datasets is the ability to group bu one of the index levels.\n",
    "#### To group by a certain level, pass the level number or name with the 'level' keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th>cty</th>\n",
       "      <th colspan=\"3\" halign=\"left\">US</th>\n",
       "      <th colspan=\"2\" halign=\"left\">JP</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tenor</th>\n",
       "      <th>1</th>\n",
       "      <th>3</th>\n",
       "      <th>5</th>\n",
       "      <th>1</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.294187</td>\n",
       "      <td>-1.912265</td>\n",
       "      <td>0.805537</td>\n",
       "      <td>1.857416</td>\n",
       "      <td>-0.511496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.460410</td>\n",
       "      <td>-0.876451</td>\n",
       "      <td>-0.180717</td>\n",
       "      <td>-0.676693</td>\n",
       "      <td>-1.873716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.881580</td>\n",
       "      <td>0.207316</td>\n",
       "      <td>-1.165858</td>\n",
       "      <td>-2.042299</td>\n",
       "      <td>0.846380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.391582</td>\n",
       "      <td>-0.171104</td>\n",
       "      <td>-0.560497</td>\n",
       "      <td>1.325864</td>\n",
       "      <td>1.096357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "cty          US                            JP          \n",
       "tenor         1         3         5         1         3\n",
       "0     -0.294187 -1.912265  0.805537  1.857416 -0.511496\n",
       "1      0.460410 -0.876451 -0.180717 -0.676693 -1.873716\n",
       "2     -1.881580  0.207316 -1.165858 -2.042299  0.846380\n",
       "3      0.391582 -0.171104 -0.560497  1.325864  1.096357"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = pd.MultiIndex.from_arrays([['US', 'US', 'US', 'JP', 'JP'],\n",
    "                                    [1, 3, 5, 1, 3]],\n",
    "                                   names=['cty', 'tenor'])\n",
    "hier_df = pd.DataFrame(np.random.randn(4,5), columns=columns)\n",
    "hier_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>cty</th>\n",
       "      <th>JP</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "cty  JP  US\n",
       "0     2   3\n",
       "1     2   3\n",
       "2     2   3\n",
       "3     2   3"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hier_df.groupby(level='cty', axis=1).count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Aggregation\n",
    "#### Aggregation refers to data transformation that produces scalar values from arrays. rg - mean, count, min, sum, etc.\n",
    "#### You can use aggregation created by you or call any method defined on the grouped object.\n",
    "#### Certain functions are not defined for groupby, but can be used with it. groupby slices up the object into pieces, calls the function for each piece and then assembles them into result object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "      <th>data1</th>\n",
       "      <th>data2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>one</td>\n",
       "      <td>1.033605</td>\n",
       "      <td>-0.208085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a</td>\n",
       "      <td>two</td>\n",
       "      <td>-0.414086</td>\n",
       "      <td>0.026467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b</td>\n",
       "      <td>one</td>\n",
       "      <td>-0.590407</td>\n",
       "      <td>0.812162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b</td>\n",
       "      <td>two</td>\n",
       "      <td>0.057372</td>\n",
       "      <td>0.365072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a</td>\n",
       "      <td>one</td>\n",
       "      <td>-0.511034</td>\n",
       "      <td>2.224576</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key1 key2     data1     data2\n",
       "0    a  one  1.033605 -0.208085\n",
       "1    a  two -0.414086  0.026467\n",
       "2    b  one -0.590407  0.812162\n",
       "3    b  two  0.057372  0.365072\n",
       "4    a  one -0.511034  2.224576"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key1\n",
       "a    0.744067\n",
       "b   -0.007406\n",
       "Name: data1, dtype: float64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped = df.groupby('key1')\n",
    "grouped['data1'].quantile(0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To use your own aggregation functions, pass it to the 'aggregate' or 'agg' method.\n",
    "#### Some methods, though not strictly aggregation functions, also work with agg.\n",
    "#### NOTE - Custom aggregation functions are much slower than optimized functions. There is some overhead (eg - function calls, data rearrangement) in constructing the intermediate group chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>data1</th>\n",
       "      <th>data2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>key1</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.544638</td>\n",
       "      <td>2.432661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>0.647779</td>\n",
       "      <td>0.447090</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         data1     data2\n",
       "key1                    \n",
       "a     1.544638  2.432661\n",
       "b     0.647779  0.447090"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def peak_to_peak(arr):\n",
    "    return arr.max() - arr.min()\n",
    "\n",
    "grouped.agg(peak_to_peak)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"8\" halign=\"left\">data1</th>\n",
       "      <th colspan=\"8\" halign=\"left\">data2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>key1</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.036162</td>\n",
       "      <td>0.865170</td>\n",
       "      <td>-0.511034</td>\n",
       "      <td>-0.462560</td>\n",
       "      <td>-0.414086</td>\n",
       "      <td>0.309759</td>\n",
       "      <td>1.033605</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.680986</td>\n",
       "      <td>1.341922</td>\n",
       "      <td>-0.208085</td>\n",
       "      <td>-0.090809</td>\n",
       "      <td>0.026467</td>\n",
       "      <td>1.125521</td>\n",
       "      <td>2.224576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>2.0</td>\n",
       "      <td>-0.266518</td>\n",
       "      <td>0.458049</td>\n",
       "      <td>-0.590407</td>\n",
       "      <td>-0.428462</td>\n",
       "      <td>-0.266518</td>\n",
       "      <td>-0.104573</td>\n",
       "      <td>0.057372</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.588617</td>\n",
       "      <td>0.316140</td>\n",
       "      <td>0.365072</td>\n",
       "      <td>0.476845</td>\n",
       "      <td>0.588617</td>\n",
       "      <td>0.700390</td>\n",
       "      <td>0.812162</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     data1                                                              \\\n",
       "     count      mean       std       min       25%       50%       75%   \n",
       "key1                                                                     \n",
       "a      3.0  0.036162  0.865170 -0.511034 -0.462560 -0.414086  0.309759   \n",
       "b      2.0 -0.266518  0.458049 -0.590407 -0.428462 -0.266518 -0.104573   \n",
       "\n",
       "               data2                                                    \\\n",
       "           max count      mean       std       min       25%       50%   \n",
       "key1                                                                     \n",
       "a     1.033605   3.0  0.680986  1.341922 -0.208085 -0.090809  0.026467   \n",
       "b     0.057372   2.0  0.588617  0.316140  0.365072  0.476845  0.588617   \n",
       "\n",
       "                          \n",
       "           75%       max  \n",
       "key1                      \n",
       "a     1.125521  2.224576  \n",
       "b     0.700390  0.812162  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Column-Wise and Multiple Function Application\n",
    "#### Aggregating a Series or columns of a DataFrame is matter of using 'aggregate' with desired function or calling a method eg - mean or std.\n",
    "#### There might be a need to aggregate using a different function based on the column, or multiple functions at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_pct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.99</td>\n",
       "      <td>1.01</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.059447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.34</td>\n",
       "      <td>1.66</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>0.160542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21.01</td>\n",
       "      <td>3.50</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>0.166587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.68</td>\n",
       "      <td>3.31</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.139780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>24.59</td>\n",
       "      <td>3.61</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>0.146808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>25.29</td>\n",
       "      <td>4.71</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>0.186240</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   total_bill   tip smoker  day    time  size   tip_pct\n",
       "0       16.99  1.01     No  Sun  Dinner     2  0.059447\n",
       "1       10.34  1.66     No  Sun  Dinner     3  0.160542\n",
       "2       21.01  3.50     No  Sun  Dinner     3  0.166587\n",
       "3       23.68  3.31     No  Sun  Dinner     2  0.139780\n",
       "4       24.59  3.61     No  Sun  Dinner     4  0.146808\n",
       "5       25.29  4.71     No  Sun  Dinner     4  0.186240"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips = pd.read_csv('examples/tips.csv')\n",
    "tips['tip_pct'] = tips['tip'] / tips['total_bill']\n",
    "\n",
    "tips[:6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For descriptive statistics, you can pass the name of function as a string.\n",
    "#### We can also pass a list of functions or function names. You will get back a DataFrame with column names from the functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "day   smoker\n",
       "Fri   No        0.151650\n",
       "      Yes       0.174783\n",
       "Sat   No        0.158048\n",
       "      Yes       0.147906\n",
       "Sun   No        0.160113\n",
       "      Yes       0.187250\n",
       "Thur  No        0.160298\n",
       "      Yes       0.163863\n",
       "Name: tip_pct, dtype: float64"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped = tips.groupby(['day', 'smoker'])\n",
    "grouped_pct = grouped['tip_pct']\n",
    "grouped_pct.agg('mean')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>peak_to_peak</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Fri</th>\n",
       "      <th>No</th>\n",
       "      <td>0.151650</td>\n",
       "      <td>0.028123</td>\n",
       "      <td>0.067349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.174783</td>\n",
       "      <td>0.051293</td>\n",
       "      <td>0.159925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sat</th>\n",
       "      <th>No</th>\n",
       "      <td>0.158048</td>\n",
       "      <td>0.039767</td>\n",
       "      <td>0.235193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.147906</td>\n",
       "      <td>0.061375</td>\n",
       "      <td>0.290095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sun</th>\n",
       "      <th>No</th>\n",
       "      <td>0.160113</td>\n",
       "      <td>0.042347</td>\n",
       "      <td>0.193226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.187250</td>\n",
       "      <td>0.154134</td>\n",
       "      <td>0.644685</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Thur</th>\n",
       "      <th>No</th>\n",
       "      <td>0.160298</td>\n",
       "      <td>0.038774</td>\n",
       "      <td>0.193350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.163863</td>\n",
       "      <td>0.039389</td>\n",
       "      <td>0.151240</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 mean       std  peak_to_peak\n",
       "day  smoker                                  \n",
       "Fri  No      0.151650  0.028123      0.067349\n",
       "     Yes     0.174783  0.051293      0.159925\n",
       "Sat  No      0.158048  0.039767      0.235193\n",
       "     Yes     0.147906  0.061375      0.290095\n",
       "Sun  No      0.160113  0.042347      0.193226\n",
       "     Yes     0.187250  0.154134      0.644685\n",
       "Thur No      0.160298  0.038774      0.193350\n",
       "     Yes     0.163863  0.039389      0.151240"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped_pct.agg(['mean', 'std', peak_to_peak])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### But you do not need to accept the names the groupby gives to the columns.\n",
    "#### lambda functions have the name \"< lambda >\" which is difficult to identify (you can check the function's __name__ attribute).\n",
    "#### If you pass a list of (name, function) tuples, first element of each tuple will used as DataFrame column names.                                                                          "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>foo</th>\n",
       "      <th>bar</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Fri</th>\n",
       "      <th>No</th>\n",
       "      <td>0.151650</td>\n",
       "      <td>0.028123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.174783</td>\n",
       "      <td>0.051293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sat</th>\n",
       "      <th>No</th>\n",
       "      <td>0.158048</td>\n",
       "      <td>0.039767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.147906</td>\n",
       "      <td>0.061375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sun</th>\n",
       "      <th>No</th>\n",
       "      <td>0.160113</td>\n",
       "      <td>0.042347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.187250</td>\n",
       "      <td>0.154134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Thur</th>\n",
       "      <th>No</th>\n",
       "      <td>0.160298</td>\n",
       "      <td>0.038774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.163863</td>\n",
       "      <td>0.039389</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  foo       bar\n",
       "day  smoker                    \n",
       "Fri  No      0.151650  0.028123\n",
       "     Yes     0.174783  0.051293\n",
       "Sat  No      0.158048  0.039767\n",
       "     Yes     0.147906  0.061375\n",
       "Sun  No      0.160113  0.042347\n",
       "     Yes     0.187250  0.154134\n",
       "Thur No      0.160298  0.038774\n",
       "     Yes     0.163863  0.039389"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped_pct.agg([('foo', 'mean'), ('bar', np.std)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You have more options with DataFrame. You can specify a list of functions to apply to all the columns or different functions per column.\n",
    "#### The resulting DataFrame will have hierarchical columns. Similar to aggregating each column seperately using 'concat' to glue results with column names in the 'keys' argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">tip_pct</th>\n",
       "      <th colspan=\"3\" halign=\"left\">total_bill</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Fri</th>\n",
       "      <th>No</th>\n",
       "      <td>4</td>\n",
       "      <td>0.151650</td>\n",
       "      <td>0.187735</td>\n",
       "      <td>4</td>\n",
       "      <td>18.420000</td>\n",
       "      <td>22.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>15</td>\n",
       "      <td>0.174783</td>\n",
       "      <td>0.263480</td>\n",
       "      <td>15</td>\n",
       "      <td>16.813333</td>\n",
       "      <td>40.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sat</th>\n",
       "      <th>No</th>\n",
       "      <td>45</td>\n",
       "      <td>0.158048</td>\n",
       "      <td>0.291990</td>\n",
       "      <td>45</td>\n",
       "      <td>19.661778</td>\n",
       "      <td>48.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>42</td>\n",
       "      <td>0.147906</td>\n",
       "      <td>0.325733</td>\n",
       "      <td>42</td>\n",
       "      <td>21.276667</td>\n",
       "      <td>50.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sun</th>\n",
       "      <th>No</th>\n",
       "      <td>57</td>\n",
       "      <td>0.160113</td>\n",
       "      <td>0.252672</td>\n",
       "      <td>57</td>\n",
       "      <td>20.506667</td>\n",
       "      <td>48.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>19</td>\n",
       "      <td>0.187250</td>\n",
       "      <td>0.710345</td>\n",
       "      <td>19</td>\n",
       "      <td>24.120000</td>\n",
       "      <td>45.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Thur</th>\n",
       "      <th>No</th>\n",
       "      <td>45</td>\n",
       "      <td>0.160298</td>\n",
       "      <td>0.266312</td>\n",
       "      <td>45</td>\n",
       "      <td>17.113111</td>\n",
       "      <td>41.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>17</td>\n",
       "      <td>0.163863</td>\n",
       "      <td>0.241255</td>\n",
       "      <td>17</td>\n",
       "      <td>19.190588</td>\n",
       "      <td>43.11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            tip_pct                     total_bill                  \n",
       "              count      mean       max      count       mean    max\n",
       "day  smoker                                                         \n",
       "Fri  No           4  0.151650  0.187735          4  18.420000  22.75\n",
       "     Yes         15  0.174783  0.263480         15  16.813333  40.17\n",
       "Sat  No          45  0.158048  0.291990         45  19.661778  48.33\n",
       "     Yes         42  0.147906  0.325733         42  21.276667  50.81\n",
       "Sun  No          57  0.160113  0.252672         57  20.506667  48.17\n",
       "     Yes         19  0.187250  0.710345         19  24.120000  45.35\n",
       "Thur No          45  0.160298  0.266312         45  17.113111  41.19\n",
       "     Yes         17  0.163863  0.241255         17  19.190588  43.11"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "functions = ['count', 'mean', 'max']\n",
    "result = grouped['tip_pct', 'total_bill'].agg(functions)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Fri</th>\n",
       "      <th>No</th>\n",
       "      <td>4</td>\n",
       "      <td>0.151650</td>\n",
       "      <td>0.187735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>15</td>\n",
       "      <td>0.174783</td>\n",
       "      <td>0.263480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sat</th>\n",
       "      <th>No</th>\n",
       "      <td>45</td>\n",
       "      <td>0.158048</td>\n",
       "      <td>0.291990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>42</td>\n",
       "      <td>0.147906</td>\n",
       "      <td>0.325733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sun</th>\n",
       "      <th>No</th>\n",
       "      <td>57</td>\n",
       "      <td>0.160113</td>\n",
       "      <td>0.252672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>19</td>\n",
       "      <td>0.187250</td>\n",
       "      <td>0.710345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Thur</th>\n",
       "      <th>No</th>\n",
       "      <td>45</td>\n",
       "      <td>0.160298</td>\n",
       "      <td>0.266312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>17</td>\n",
       "      <td>0.163863</td>\n",
       "      <td>0.241255</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             count      mean       max\n",
       "day  smoker                           \n",
       "Fri  No          4  0.151650  0.187735\n",
       "     Yes        15  0.174783  0.263480\n",
       "Sat  No         45  0.158048  0.291990\n",
       "     Yes        42  0.147906  0.325733\n",
       "Sun  No         57  0.160113  0.252672\n",
       "     Yes        19  0.187250  0.710345\n",
       "Thur No         45  0.160298  0.266312\n",
       "     Yes        17  0.163863  0.241255"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result['tip_pct']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can also use list of tuples to have custom names for result columns, just like in Series.\n",
    "#### You can also apply different functions to one or more columns bu passing a dict containing a mapping of column names with function specifications to 'agg'.\n",
    "#### A DataFrame will have hierarchical columns only if multiple functions are applied to atleast one column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">tip_pct</th>\n",
       "      <th colspan=\"2\" halign=\"left\">total_bill</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Durchschnitt</th>\n",
       "      <th>Abweichung</th>\n",
       "      <th>Durchschnitt</th>\n",
       "      <th>Abweichung</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Fri</th>\n",
       "      <th>No</th>\n",
       "      <td>0.151650</td>\n",
       "      <td>0.000791</td>\n",
       "      <td>18.420000</td>\n",
       "      <td>25.596333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.174783</td>\n",
       "      <td>0.002631</td>\n",
       "      <td>16.813333</td>\n",
       "      <td>82.562438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sat</th>\n",
       "      <th>No</th>\n",
       "      <td>0.158048</td>\n",
       "      <td>0.001581</td>\n",
       "      <td>19.661778</td>\n",
       "      <td>79.908965</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.147906</td>\n",
       "      <td>0.003767</td>\n",
       "      <td>21.276667</td>\n",
       "      <td>101.387535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sun</th>\n",
       "      <th>No</th>\n",
       "      <td>0.160113</td>\n",
       "      <td>0.001793</td>\n",
       "      <td>20.506667</td>\n",
       "      <td>66.099980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.187250</td>\n",
       "      <td>0.023757</td>\n",
       "      <td>24.120000</td>\n",
       "      <td>109.046044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Thur</th>\n",
       "      <th>No</th>\n",
       "      <td>0.160298</td>\n",
       "      <td>0.001503</td>\n",
       "      <td>17.113111</td>\n",
       "      <td>59.625081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.163863</td>\n",
       "      <td>0.001551</td>\n",
       "      <td>19.190588</td>\n",
       "      <td>69.808518</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 tip_pct              total_bill            \n",
       "            Durchschnitt Abweichung Durchschnitt  Abweichung\n",
       "day  smoker                                                 \n",
       "Fri  No         0.151650   0.000791    18.420000   25.596333\n",
       "     Yes        0.174783   0.002631    16.813333   82.562438\n",
       "Sat  No         0.158048   0.001581    19.661778   79.908965\n",
       "     Yes        0.147906   0.003767    21.276667  101.387535\n",
       "Sun  No         0.160113   0.001793    20.506667   66.099980\n",
       "     Yes        0.187250   0.023757    24.120000  109.046044\n",
       "Thur No         0.160298   0.001503    17.113111   59.625081\n",
       "     Yes        0.163863   0.001551    19.190588   69.808518"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ftuples = [('Durchschnitt', 'mean'), ('Abweichung', np.var)]\n",
    "\n",
    "grouped['tip_pct', 'total_bill'].agg(ftuples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>tip</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Fri</th>\n",
       "      <th>No</th>\n",
       "      <td>3.50</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>4.73</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sat</th>\n",
       "      <th>No</th>\n",
       "      <td>9.00</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>10.00</td>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sun</th>\n",
       "      <th>No</th>\n",
       "      <td>6.00</td>\n",
       "      <td>167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>6.50</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Thur</th>\n",
       "      <th>No</th>\n",
       "      <td>6.70</td>\n",
       "      <td>112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>5.00</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               tip  size\n",
       "day  smoker             \n",
       "Fri  No       3.50     9\n",
       "     Yes      4.73    31\n",
       "Sat  No       9.00   115\n",
       "     Yes     10.00   104\n",
       "Sun  No       6.00   167\n",
       "     Yes      6.50    49\n",
       "Thur No       6.70   112\n",
       "     Yes      5.00    40"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.agg({'tip': np.max, 'size': 'sum'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"4\" halign=\"left\">tip_pct</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>sum</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Fri</th>\n",
       "      <th>No</th>\n",
       "      <td>0.120385</td>\n",
       "      <td>0.187735</td>\n",
       "      <td>0.151650</td>\n",
       "      <td>0.028123</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.103555</td>\n",
       "      <td>0.263480</td>\n",
       "      <td>0.174783</td>\n",
       "      <td>0.051293</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sat</th>\n",
       "      <th>No</th>\n",
       "      <td>0.056797</td>\n",
       "      <td>0.291990</td>\n",
       "      <td>0.158048</td>\n",
       "      <td>0.039767</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.035638</td>\n",
       "      <td>0.325733</td>\n",
       "      <td>0.147906</td>\n",
       "      <td>0.061375</td>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sun</th>\n",
       "      <th>No</th>\n",
       "      <td>0.059447</td>\n",
       "      <td>0.252672</td>\n",
       "      <td>0.160113</td>\n",
       "      <td>0.042347</td>\n",
       "      <td>167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.065660</td>\n",
       "      <td>0.710345</td>\n",
       "      <td>0.187250</td>\n",
       "      <td>0.154134</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Thur</th>\n",
       "      <th>No</th>\n",
       "      <td>0.072961</td>\n",
       "      <td>0.266312</td>\n",
       "      <td>0.160298</td>\n",
       "      <td>0.038774</td>\n",
       "      <td>112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.090014</td>\n",
       "      <td>0.241255</td>\n",
       "      <td>0.163863</td>\n",
       "      <td>0.039389</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              tip_pct                               size\n",
       "                  min       max      mean       std  sum\n",
       "day  smoker                                             \n",
       "Fri  No      0.120385  0.187735  0.151650  0.028123    9\n",
       "     Yes     0.103555  0.263480  0.174783  0.051293   31\n",
       "Sat  No      0.056797  0.291990  0.158048  0.039767  115\n",
       "     Yes     0.035638  0.325733  0.147906  0.061375  104\n",
       "Sun  No      0.059447  0.252672  0.160113  0.042347  167\n",
       "     Yes     0.065660  0.710345  0.187250  0.154134   49\n",
       "Thur No      0.072961  0.266312  0.160298  0.038774  112\n",
       "     Yes     0.090014  0.241255  0.163863  0.039389   40"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.agg({'tip_pct': ['min', 'max', 'mean', 'std'],\n",
    "            'size': 'sum'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Returning Aggregated Data without Row Indexes\n",
    "#### Usually aggregated data comes back with an index (mostly hierarchical), composed of unique group key combinations.\n",
    "#### This is not always desirable. we can disable this by passing 'as_index=False' to 'groupby'.\n",
    "#### We can also use 'reset_index' on the result to get same format, but the former method avoids unnecessary computations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_pct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fri</td>\n",
       "      <td>No</td>\n",
       "      <td>18.420000</td>\n",
       "      <td>2.812500</td>\n",
       "      <td>2.250000</td>\n",
       "      <td>0.151650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fri</td>\n",
       "      <td>Yes</td>\n",
       "      <td>16.813333</td>\n",
       "      <td>2.714000</td>\n",
       "      <td>2.066667</td>\n",
       "      <td>0.174783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Sat</td>\n",
       "      <td>No</td>\n",
       "      <td>19.661778</td>\n",
       "      <td>3.102889</td>\n",
       "      <td>2.555556</td>\n",
       "      <td>0.158048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sat</td>\n",
       "      <td>Yes</td>\n",
       "      <td>21.276667</td>\n",
       "      <td>2.875476</td>\n",
       "      <td>2.476190</td>\n",
       "      <td>0.147906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sun</td>\n",
       "      <td>No</td>\n",
       "      <td>20.506667</td>\n",
       "      <td>3.167895</td>\n",
       "      <td>2.929825</td>\n",
       "      <td>0.160113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Sun</td>\n",
       "      <td>Yes</td>\n",
       "      <td>24.120000</td>\n",
       "      <td>3.516842</td>\n",
       "      <td>2.578947</td>\n",
       "      <td>0.187250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Thur</td>\n",
       "      <td>No</td>\n",
       "      <td>17.113111</td>\n",
       "      <td>2.673778</td>\n",
       "      <td>2.488889</td>\n",
       "      <td>0.160298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Thur</td>\n",
       "      <td>Yes</td>\n",
       "      <td>19.190588</td>\n",
       "      <td>3.030000</td>\n",
       "      <td>2.352941</td>\n",
       "      <td>0.163863</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    day smoker  total_bill       tip      size   tip_pct\n",
       "0   Fri     No   18.420000  2.812500  2.250000  0.151650\n",
       "1   Fri    Yes   16.813333  2.714000  2.066667  0.174783\n",
       "2   Sat     No   19.661778  3.102889  2.555556  0.158048\n",
       "3   Sat    Yes   21.276667  2.875476  2.476190  0.147906\n",
       "4   Sun     No   20.506667  3.167895  2.929825  0.160113\n",
       "5   Sun    Yes   24.120000  3.516842  2.578947  0.187250\n",
       "6  Thur     No   17.113111  2.673778  2.488889  0.160298\n",
       "7  Thur    Yes   19.190588  3.030000  2.352941  0.163863"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.groupby(['day', 'smoker'], as_index=False).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply: General split-apply-combine\n",
    "#### The most general-purpose groupby method is 'apply'. It splits and object into pieces, invokes passed function on each piece and then concatenates the pieces together.\n",
    "#### when we use groupby with a function, the function is called on each row of the DataFrame and the results are glued together using 'pandas.concat', labeling the pieces with group names. The result has a hierarchical index whose inner level has index values from original DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_pct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>14.31</td>\n",
       "      <td>4.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.279525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>23.17</td>\n",
       "      <td>6.50</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>0.280535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>232</th>\n",
       "      <td>11.61</td>\n",
       "      <td>3.39</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.291990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>3.07</td>\n",
       "      <td>1.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>1</td>\n",
       "      <td>0.325733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>9.60</td>\n",
       "      <td>4.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.416667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>7.25</td>\n",
       "      <td>5.15</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.710345</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     total_bill   tip smoker  day    time  size   tip_pct\n",
       "109       14.31  4.00    Yes  Sat  Dinner     2  0.279525\n",
       "183       23.17  6.50    Yes  Sun  Dinner     4  0.280535\n",
       "232       11.61  3.39     No  Sat  Dinner     2  0.291990\n",
       "67         3.07  1.00    Yes  Sat  Dinner     1  0.325733\n",
       "178        9.60  4.00    Yes  Sun  Dinner     2  0.416667\n",
       "172        7.25  5.15    Yes  Sun  Dinner     2  0.710345"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def top(df, n=5, column='tip_pct'):\n",
    "    return df.sort_values(by=column)[-n:]\n",
    "\n",
    "\n",
    "top(tips, n=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_pct</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">No</th>\n",
       "      <th>88</th>\n",
       "      <td>24.71</td>\n",
       "      <td>5.85</td>\n",
       "      <td>No</td>\n",
       "      <td>Thur</td>\n",
       "      <td>Lunch</td>\n",
       "      <td>2</td>\n",
       "      <td>0.236746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>20.69</td>\n",
       "      <td>5.00</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>5</td>\n",
       "      <td>0.241663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>10.29</td>\n",
       "      <td>2.60</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.252672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>7.51</td>\n",
       "      <td>2.00</td>\n",
       "      <td>No</td>\n",
       "      <td>Thur</td>\n",
       "      <td>Lunch</td>\n",
       "      <td>2</td>\n",
       "      <td>0.266312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>232</th>\n",
       "      <td>11.61</td>\n",
       "      <td>3.39</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.291990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">Yes</th>\n",
       "      <th>109</th>\n",
       "      <td>14.31</td>\n",
       "      <td>4.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.279525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>23.17</td>\n",
       "      <td>6.50</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>0.280535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>3.07</td>\n",
       "      <td>1.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>1</td>\n",
       "      <td>0.325733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>9.60</td>\n",
       "      <td>4.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.416667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>7.25</td>\n",
       "      <td>5.15</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.710345</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            total_bill   tip smoker   day    time  size   tip_pct\n",
       "smoker                                                           \n",
       "No     88        24.71  5.85     No  Thur   Lunch     2  0.236746\n",
       "       185       20.69  5.00     No   Sun  Dinner     5  0.241663\n",
       "       51        10.29  2.60     No   Sun  Dinner     2  0.252672\n",
       "       149        7.51  2.00     No  Thur   Lunch     2  0.266312\n",
       "       232       11.61  3.39     No   Sat  Dinner     2  0.291990\n",
       "Yes    109       14.31  4.00    Yes   Sat  Dinner     2  0.279525\n",
       "       183       23.17  6.50    Yes   Sun  Dinner     4  0.280535\n",
       "       67         3.07  1.00    Yes   Sat  Dinner     1  0.325733\n",
       "       178        9.60  4.00    Yes   Sun  Dinner     2  0.416667\n",
       "       172        7.25  5.15    Yes   Sun  Dinner     2  0.710345"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.groupby('smoker').apply(top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If you want to pass a function to apply with other arguments or keywords, you can pass them after the function.\n",
    "#### NOTE - Beside basic usage, it requires creativity to use apply and get the most out of it. What happens inside a function depends on you, as long as the function returns a pandas object or scalar value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_pct</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">No</th>\n",
       "      <th>Fri</th>\n",
       "      <th>94</th>\n",
       "      <td>22.75</td>\n",
       "      <td>3.25</td>\n",
       "      <td>No</td>\n",
       "      <td>Fri</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sat</th>\n",
       "      <th>212</th>\n",
       "      <td>48.33</td>\n",
       "      <td>9.00</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>0.186220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sun</th>\n",
       "      <th>156</th>\n",
       "      <td>48.17</td>\n",
       "      <td>5.00</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>6</td>\n",
       "      <td>0.103799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thur</th>\n",
       "      <th>142</th>\n",
       "      <td>41.19</td>\n",
       "      <td>5.00</td>\n",
       "      <td>No</td>\n",
       "      <td>Thur</td>\n",
       "      <td>Lunch</td>\n",
       "      <td>5</td>\n",
       "      <td>0.121389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">Yes</th>\n",
       "      <th>Fri</th>\n",
       "      <th>95</th>\n",
       "      <td>40.17</td>\n",
       "      <td>4.73</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fri</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>0.117750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sat</th>\n",
       "      <th>170</th>\n",
       "      <td>50.81</td>\n",
       "      <td>10.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>0.196812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sun</th>\n",
       "      <th>182</th>\n",
       "      <td>45.35</td>\n",
       "      <td>3.50</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>0.077178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thur</th>\n",
       "      <th>197</th>\n",
       "      <td>43.11</td>\n",
       "      <td>5.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Thur</td>\n",
       "      <td>Lunch</td>\n",
       "      <td>4</td>\n",
       "      <td>0.115982</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 total_bill    tip smoker   day    time  size   tip_pct\n",
       "smoker day                                                             \n",
       "No     Fri  94        22.75   3.25     No   Fri  Dinner     2  0.142857\n",
       "       Sat  212       48.33   9.00     No   Sat  Dinner     4  0.186220\n",
       "       Sun  156       48.17   5.00     No   Sun  Dinner     6  0.103799\n",
       "       Thur 142       41.19   5.00     No  Thur   Lunch     5  0.121389\n",
       "Yes    Fri  95        40.17   4.73    Yes   Fri  Dinner     4  0.117750\n",
       "       Sat  170       50.81  10.00    Yes   Sat  Dinner     3  0.196812\n",
       "       Sun  182       45.35   3.50    Yes   Sun  Dinner     3  0.077178\n",
       "       Thur 197       43.11   5.00    Yes  Thur   Lunch     4  0.115982"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.groupby(['smoker', 'day']).apply(top, n=1, column='total_bill')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inside groupby, when you want to invoke a method, there is a shortcut for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>No</th>\n",
       "      <td>151.0</td>\n",
       "      <td>0.159328</td>\n",
       "      <td>0.039910</td>\n",
       "      <td>0.056797</td>\n",
       "      <td>0.136906</td>\n",
       "      <td>0.155625</td>\n",
       "      <td>0.185014</td>\n",
       "      <td>0.291990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>93.0</td>\n",
       "      <td>0.163196</td>\n",
       "      <td>0.085119</td>\n",
       "      <td>0.035638</td>\n",
       "      <td>0.106771</td>\n",
       "      <td>0.153846</td>\n",
       "      <td>0.195059</td>\n",
       "      <td>0.710345</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        count      mean       std       min       25%       50%       75%  \\\n",
       "smoker                                                                      \n",
       "No      151.0  0.159328  0.039910  0.056797  0.136906  0.155625  0.185014   \n",
       "Yes      93.0  0.163196  0.085119  0.035638  0.106771  0.153846  0.195059   \n",
       "\n",
       "             max  \n",
       "smoker            \n",
       "No      0.291990  \n",
       "Yes     0.710345  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Normal route\n",
    "result = tips.groupby('smoker')['tip_pct'].describe()\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "       smoker\n",
       "count  No        151.000000\n",
       "       Yes        93.000000\n",
       "mean   No          0.159328\n",
       "       Yes         0.163196\n",
       "std    No          0.039910\n",
       "       Yes         0.085119\n",
       "min    No          0.056797\n",
       "       Yes         0.035638\n",
       "25%    No          0.136906\n",
       "       Yes         0.106771\n",
       "50%    No          0.155625\n",
       "       Yes         0.153846\n",
       "75%    No          0.185014\n",
       "       Yes         0.195059\n",
       "max    No          0.291990\n",
       "       Yes         0.710345\n",
       "dtype: float64"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.unstack('smoker')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_pct</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"16\" valign=\"top\">Fri</th>\n",
       "      <th rowspan=\"8\" valign=\"top\">No</th>\n",
       "      <th>count</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>18.420000</td>\n",
       "      <td>2.812500</td>\n",
       "      <td>2.250000</td>\n",
       "      <td>0.151650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>5.059282</td>\n",
       "      <td>0.898494</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.028123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>12.460000</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.120385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>15.100000</td>\n",
       "      <td>2.625000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.137239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>19.235000</td>\n",
       "      <td>3.125000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.149241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>22.555000</td>\n",
       "      <td>3.312500</td>\n",
       "      <td>2.250000</td>\n",
       "      <td>0.163652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>22.750000</td>\n",
       "      <td>3.500000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.187735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"8\" valign=\"top\">Yes</th>\n",
       "      <th>count</th>\n",
       "      <td>15.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>15.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>16.813333</td>\n",
       "      <td>2.714000</td>\n",
       "      <td>2.066667</td>\n",
       "      <td>0.174783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.086388</td>\n",
       "      <td>1.077668</td>\n",
       "      <td>0.593617</td>\n",
       "      <td>0.051293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>5.750000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.103555</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>11.690000</td>\n",
       "      <td>1.960000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.133739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>13.420000</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.173913</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>18.665000</td>\n",
       "      <td>3.240000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.209240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>40.170000</td>\n",
       "      <td>4.730000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.263480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"14\" valign=\"top\">Sat</th>\n",
       "      <th rowspan=\"8\" valign=\"top\">No</th>\n",
       "      <th>count</th>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>19.661778</td>\n",
       "      <td>3.102889</td>\n",
       "      <td>2.555556</td>\n",
       "      <td>0.158048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>8.939181</td>\n",
       "      <td>1.642088</td>\n",
       "      <td>0.784960</td>\n",
       "      <td>0.039767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>7.250000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.056797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>14.730000</td>\n",
       "      <td>2.010000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.136240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>17.820000</td>\n",
       "      <td>2.750000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.150152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>20.650000</td>\n",
       "      <td>3.390000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.183915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>48.330000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.291990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"6\" valign=\"top\">Yes</th>\n",
       "      <th>count</th>\n",
       "      <td>42.000000</td>\n",
       "      <td>42.000000</td>\n",
       "      <td>42.000000</td>\n",
       "      <td>42.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>21.276667</td>\n",
       "      <td>2.875476</td>\n",
       "      <td>2.476190</td>\n",
       "      <td>0.147906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>10.069138</td>\n",
       "      <td>1.630580</td>\n",
       "      <td>0.862161</td>\n",
       "      <td>0.061375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>3.070000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.035638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>13.405000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.091797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>20.390000</td>\n",
       "      <td>2.690000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.153624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"14\" valign=\"top\">Sun</th>\n",
       "      <th rowspan=\"6\" valign=\"top\">No</th>\n",
       "      <th>std</th>\n",
       "      <td>8.130189</td>\n",
       "      <td>1.224785</td>\n",
       "      <td>1.032674</td>\n",
       "      <td>0.042347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>8.770000</td>\n",
       "      <td>1.010000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.059447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>14.780000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.139780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>18.430000</td>\n",
       "      <td>3.020000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.161665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>25.000000</td>\n",
       "      <td>3.920000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.185185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>48.170000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>0.252672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"8\" valign=\"top\">Yes</th>\n",
       "      <th>count</th>\n",
       "      <td>19.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>19.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>24.120000</td>\n",
       "      <td>3.516842</td>\n",
       "      <td>2.578947</td>\n",
       "      <td>0.187250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>10.442511</td>\n",
       "      <td>1.261151</td>\n",
       "      <td>0.901591</td>\n",
       "      <td>0.154134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>7.250000</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.065660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>17.165000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.097723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>23.100000</td>\n",
       "      <td>3.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.138122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>32.375000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.215325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>45.350000</td>\n",
       "      <td>6.500000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.710345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"16\" valign=\"top\">Thur</th>\n",
       "      <th rowspan=\"8\" valign=\"top\">No</th>\n",
       "      <th>count</th>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>17.113111</td>\n",
       "      <td>2.673778</td>\n",
       "      <td>2.488889</td>\n",
       "      <td>0.160298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>7.721728</td>\n",
       "      <td>1.282964</td>\n",
       "      <td>1.179796</td>\n",
       "      <td>0.038774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>7.510000</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.072961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>11.690000</td>\n",
       "      <td>1.800000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.137741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>15.950000</td>\n",
       "      <td>2.180000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.153492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>20.270000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.184843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>41.190000</td>\n",
       "      <td>6.700000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>0.266312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"8\" valign=\"top\">Yes</th>\n",
       "      <th>count</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>17.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>19.190588</td>\n",
       "      <td>3.030000</td>\n",
       "      <td>2.352941</td>\n",
       "      <td>0.163863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>8.355149</td>\n",
       "      <td>1.113491</td>\n",
       "      <td>0.701888</td>\n",
       "      <td>0.039389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>10.340000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.090014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>13.510000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.148038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>16.470000</td>\n",
       "      <td>2.560000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.153846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>19.810000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.194837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>43.110000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.241255</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>64 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   total_bill        tip       size    tip_pct\n",
       "day  smoker                                                   \n",
       "Fri  No     count    4.000000   4.000000   4.000000   4.000000\n",
       "            mean    18.420000   2.812500   2.250000   0.151650\n",
       "            std      5.059282   0.898494   0.500000   0.028123\n",
       "            min     12.460000   1.500000   2.000000   0.120385\n",
       "            25%     15.100000   2.625000   2.000000   0.137239\n",
       "            50%     19.235000   3.125000   2.000000   0.149241\n",
       "            75%     22.555000   3.312500   2.250000   0.163652\n",
       "            max     22.750000   3.500000   3.000000   0.187735\n",
       "     Yes    count   15.000000  15.000000  15.000000  15.000000\n",
       "            mean    16.813333   2.714000   2.066667   0.174783\n",
       "            std      9.086388   1.077668   0.593617   0.051293\n",
       "            min      5.750000   1.000000   1.000000   0.103555\n",
       "            25%     11.690000   1.960000   2.000000   0.133739\n",
       "            50%     13.420000   2.500000   2.000000   0.173913\n",
       "            75%     18.665000   3.240000   2.000000   0.209240\n",
       "            max     40.170000   4.730000   4.000000   0.263480\n",
       "Sat  No     count   45.000000  45.000000  45.000000  45.000000\n",
       "            mean    19.661778   3.102889   2.555556   0.158048\n",
       "            std      8.939181   1.642088   0.784960   0.039767\n",
       "            min      7.250000   1.000000   1.000000   0.056797\n",
       "            25%     14.730000   2.010000   2.000000   0.136240\n",
       "            50%     17.820000   2.750000   2.000000   0.150152\n",
       "            75%     20.650000   3.390000   3.000000   0.183915\n",
       "            max     48.330000   9.000000   4.000000   0.291990\n",
       "     Yes    count   42.000000  42.000000  42.000000  42.000000\n",
       "            mean    21.276667   2.875476   2.476190   0.147906\n",
       "            std     10.069138   1.630580   0.862161   0.061375\n",
       "            min      3.070000   1.000000   1.000000   0.035638\n",
       "            25%     13.405000   2.000000   2.000000   0.091797\n",
       "            50%     20.390000   2.690000   2.000000   0.153624\n",
       "...                       ...        ...        ...        ...\n",
       "Sun  No     std      8.130189   1.224785   1.032674   0.042347\n",
       "            min      8.770000   1.010000   2.000000   0.059447\n",
       "            25%     14.780000   2.000000   2.000000   0.139780\n",
       "            50%     18.430000   3.020000   3.000000   0.161665\n",
       "            75%     25.000000   3.920000   4.000000   0.185185\n",
       "            max     48.170000   6.000000   6.000000   0.252672\n",
       "     Yes    count   19.000000  19.000000  19.000000  19.000000\n",
       "            mean    24.120000   3.516842   2.578947   0.187250\n",
       "            std     10.442511   1.261151   0.901591   0.154134\n",
       "            min      7.250000   1.500000   2.000000   0.065660\n",
       "            25%     17.165000   3.000000   2.000000   0.097723\n",
       "            50%     23.100000   3.500000   2.000000   0.138122\n",
       "            75%     32.375000   4.000000   3.000000   0.215325\n",
       "            max     45.350000   6.500000   5.000000   0.710345\n",
       "Thur No     count   45.000000  45.000000  45.000000  45.000000\n",
       "            mean    17.113111   2.673778   2.488889   0.160298\n",
       "            std      7.721728   1.282964   1.179796   0.038774\n",
       "            min      7.510000   1.250000   1.000000   0.072961\n",
       "            25%     11.690000   1.800000   2.000000   0.137741\n",
       "            50%     15.950000   2.180000   2.000000   0.153492\n",
       "            75%     20.270000   3.000000   2.000000   0.184843\n",
       "            max     41.190000   6.700000   6.000000   0.266312\n",
       "     Yes    count   17.000000  17.000000  17.000000  17.000000\n",
       "            mean    19.190588   3.030000   2.352941   0.163863\n",
       "            std      8.355149   1.113491   0.701888   0.039389\n",
       "            min     10.340000   2.000000   2.000000   0.090014\n",
       "            25%     13.510000   2.000000   2.000000   0.148038\n",
       "            50%     16.470000   2.560000   2.000000   0.153846\n",
       "            75%     19.810000   4.000000   2.000000   0.194837\n",
       "            max     43.110000   5.000000   4.000000   0.241255\n",
       "\n",
       "[64 rows x 4 columns]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shortcut\n",
    "\n",
    "f = lambda x: x.describe()\n",
    "grouped.apply(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Suppressing the Group Keys\n",
    "#### The result from groupby has been an object with hierarchical index fromed by group keys along with indexes of each piece of original object.\n",
    "#### We can disable this by passing 'group_keys=False' to groupby."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_pct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>24.71</td>\n",
       "      <td>5.85</td>\n",
       "      <td>No</td>\n",
       "      <td>Thur</td>\n",
       "      <td>Lunch</td>\n",
       "      <td>2</td>\n",
       "      <td>0.236746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>20.69</td>\n",
       "      <td>5.00</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>5</td>\n",
       "      <td>0.241663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>10.29</td>\n",
       "      <td>2.60</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.252672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>7.51</td>\n",
       "      <td>2.00</td>\n",
       "      <td>No</td>\n",
       "      <td>Thur</td>\n",
       "      <td>Lunch</td>\n",
       "      <td>2</td>\n",
       "      <td>0.266312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>232</th>\n",
       "      <td>11.61</td>\n",
       "      <td>3.39</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.291990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>14.31</td>\n",
       "      <td>4.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.279525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>23.17</td>\n",
       "      <td>6.50</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>0.280535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>3.07</td>\n",
       "      <td>1.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>1</td>\n",
       "      <td>0.325733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>9.60</td>\n",
       "      <td>4.00</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.416667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>7.25</td>\n",
       "      <td>5.15</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.710345</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     total_bill   tip smoker   day    time  size   tip_pct\n",
       "88        24.71  5.85     No  Thur   Lunch     2  0.236746\n",
       "185       20.69  5.00     No   Sun  Dinner     5  0.241663\n",
       "51        10.29  2.60     No   Sun  Dinner     2  0.252672\n",
       "149        7.51  2.00     No  Thur   Lunch     2  0.266312\n",
       "232       11.61  3.39     No   Sat  Dinner     2  0.291990\n",
       "109       14.31  4.00    Yes   Sat  Dinner     2  0.279525\n",
       "183       23.17  6.50    Yes   Sun  Dinner     4  0.280535\n",
       "67         3.07  1.00    Yes   Sat  Dinner     1  0.325733\n",
       "178        9.60  4.00    Yes   Sun  Dinner     2  0.416667\n",
       "172        7.25  5.15    Yes   Sun  Dinner     2  0.710345"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.groupby('smoker', group_keys=False).apply(top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quantile and Bucket Analysis\n",
    "#### pandas has some tools for slicing data into buckets with bins of your choosing or sample quantiles. eg - 'cut' and 'qcut'.\n",
    "#### Combining these functions with groupby makes it convinient to eprform bucket or quantile analysis on a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     (-3.283, -1.668]\n",
       "1     (-0.0593, 1.549]\n",
       "2    (-1.668, -0.0593]\n",
       "3     (-0.0593, 1.549]\n",
       "4    (-1.668, -0.0593]\n",
       "5    (-1.668, -0.0593]\n",
       "6     (-0.0593, 1.549]\n",
       "7     (-0.0593, 1.549]\n",
       "8     (-0.0593, 1.549]\n",
       "9    (-1.668, -0.0593]\n",
       "Name: data1, dtype: category\n",
       "Categories (4, interval[float64]): [(-3.283, -1.668] < (-1.668, -0.0593] < (-0.0593, 1.549] < (1.549, 3.158]]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "frame = pd.DataFrame({'data1': np.random.randn(1000),\n",
    "                     'data2': np.random.randn(1000)})\n",
    "\n",
    "quartiles = pd.cut(frame.data1, 4)\n",
    "quartiles[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The categorical object returned by cut can be passed directly to groupby to compute set of statistics for any column.\n",
    "#### For equal-sized buckets based on sample quantiles, we can use 'qcut'. We can pass 'labels=False' to get quantile numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "      <th>min</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>data1</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>(-3.283, -1.668]</th>\n",
       "      <td>52.0</td>\n",
       "      <td>3.164256</td>\n",
       "      <td>0.128052</td>\n",
       "      <td>-2.280419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(-1.668, -0.0593]</th>\n",
       "      <td>457.0</td>\n",
       "      <td>3.152965</td>\n",
       "      <td>-0.044853</td>\n",
       "      <td>-3.462515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(-0.0593, 1.549]</th>\n",
       "      <td>433.0</td>\n",
       "      <td>3.836951</td>\n",
       "      <td>-0.066945</td>\n",
       "      <td>-3.237170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(1.549, 3.158]</th>\n",
       "      <td>58.0</td>\n",
       "      <td>2.245450</td>\n",
       "      <td>0.111512</td>\n",
       "      <td>-1.798201</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   count       max      mean       min\n",
       "data1                                                 \n",
       "(-3.283, -1.668]    52.0  3.164256  0.128052 -2.280419\n",
       "(-1.668, -0.0593]  457.0  3.152965 -0.044853 -3.462515\n",
       "(-0.0593, 1.549]   433.0  3.836951 -0.066945 -3.237170\n",
       "(1.549, 3.158]      58.0  2.245450  0.111512 -1.798201"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_stats(group):\n",
    "    return {'min': group.min(), 'max': group.max(),\n",
    "           'count': group.count(), 'mean': group.mean()}\n",
    "\n",
    "grouped = frame.data2.groupby(quartiles)\n",
    "grouped.apply(get_stats).unstack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "      <th>min</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>data1</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100.0</td>\n",
       "      <td>3.164256</td>\n",
       "      <td>0.043477</td>\n",
       "      <td>-2.496022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100.0</td>\n",
       "      <td>2.135533</td>\n",
       "      <td>-0.170752</td>\n",
       "      <td>-3.016819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100.0</td>\n",
       "      <td>2.979919</td>\n",
       "      <td>0.106324</td>\n",
       "      <td>-3.462515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100.0</td>\n",
       "      <td>1.873858</td>\n",
       "      <td>-0.126911</td>\n",
       "      <td>-2.408925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100.0</td>\n",
       "      <td>2.533027</td>\n",
       "      <td>-0.005920</td>\n",
       "      <td>-2.369702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>100.0</td>\n",
       "      <td>3.836951</td>\n",
       "      <td>0.059023</td>\n",
       "      <td>-3.237170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>100.0</td>\n",
       "      <td>2.740522</td>\n",
       "      <td>-0.206760</td>\n",
       "      <td>-3.154231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>100.0</td>\n",
       "      <td>2.742002</td>\n",
       "      <td>-0.023638</td>\n",
       "      <td>-2.201357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>100.0</td>\n",
       "      <td>2.309644</td>\n",
       "      <td>-0.017650</td>\n",
       "      <td>-2.028481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>100.0</td>\n",
       "      <td>2.245450</td>\n",
       "      <td>-0.020779</td>\n",
       "      <td>-2.067576</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       count       max      mean       min\n",
       "data1                                     \n",
       "0      100.0  3.164256  0.043477 -2.496022\n",
       "1      100.0  2.135533 -0.170752 -3.016819\n",
       "2      100.0  2.979919  0.106324 -3.462515\n",
       "3      100.0  1.873858 -0.126911 -2.408925\n",
       "4      100.0  2.533027 -0.005920 -2.369702\n",
       "5      100.0  3.836951  0.059023 -3.237170\n",
       "6      100.0  2.740522 -0.206760 -3.154231\n",
       "7      100.0  2.742002 -0.023638 -2.201357\n",
       "8      100.0  2.309644 -0.017650 -2.028481\n",
       "9      100.0  2.245450 -0.020779 -2.067576"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouping = pd.qcut(frame.data1, 10, labels=False)\n",
    "grouped = frame.data2.groupby(grouping)\n",
    "\n",
    "grouped.apply(get_stats).unstack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Filling Missing Values with Group-Specific Values\n",
    "#### When cleaning up missing data, you can drop the missing columns or rows using 'dropna'. But you may also want to impute (fill in) the null values using a fixed or derived value using 'fillna'.\n",
    "#### To fill a value varying by group, you can group the data and use 'apply' with a function on each of the data chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         NaN\n",
       "1    0.245974\n",
       "2         NaN\n",
       "3   -0.617814\n",
       "4         NaN\n",
       "5    0.616995\n",
       "dtype: float64"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(np.random.randn(6))\n",
    "s[::2] = np.nan\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.081718\n",
       "1    0.245974\n",
       "2    0.081718\n",
       "3   -0.617814\n",
       "4    0.081718\n",
       "5    0.616995\n",
       "dtype: float64"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.fillna(s.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ohio          1.891586\n",
       "New York      0.408207\n",
       "Vermont       1.604762\n",
       "Florida       0.036644\n",
       "Oregon       -0.255685\n",
       "Nevada        1.689558\n",
       "California   -0.434870\n",
       "Idaho        -0.577308\n",
       "dtype: float64"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states = ['Ohio', 'New York', 'Vermont', 'Florida',\n",
    "         'Oregon', 'Nevada', 'California', 'Idaho']\n",
    "\n",
    "# Create 4 copies of elements in both East and West as key\n",
    "group_key = ['East'] * 4 + ['West'] * 4\n",
    "data = pd.Series(np.random.randn(8), index=states)\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ohio          1.891586\n",
       "New York      0.408207\n",
       "Vermont            NaN\n",
       "Florida       0.036644\n",
       "Oregon       -0.255685\n",
       "Nevada             NaN\n",
       "California   -0.434870\n",
       "Idaho              NaN\n",
       "dtype: float64"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# setting some values to NA\n",
    "data[['Vermont', 'Nevada', 'Idaho']] = np.nan\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "East    0.778812\n",
       "West   -0.345278\n",
       "dtype: float64"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.groupby(group_key).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ohio          1.891586\n",
       "New York      0.408207\n",
       "Vermont       0.778812\n",
       "Florida       0.036644\n",
       "Oregon       -0.255685\n",
       "Nevada       -0.345278\n",
       "California   -0.434870\n",
       "Idaho        -0.345278\n",
       "dtype: float64"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# setting the NA values with group means\n",
    "fill_mean = lambda g: g.fillna(g.mean())\n",
    "data.groupby(group_key).apply(fill_mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You may even have predeefined fill values by group. We can use the name attribute in the groups to our advantage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ohio          1.891586\n",
       "New York      0.408207\n",
       "Vermont       0.500000\n",
       "Florida       0.036644\n",
       "Oregon       -0.255685\n",
       "Nevada       -1.000000\n",
       "California   -0.434870\n",
       "Idaho        -1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fill_values = {'East': 0.5, 'West': -1}\n",
    "fill_func = lambda g: g.fillna(fill_values[g.name])\n",
    "\n",
    "data.groupby(group_key).apply(fill_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Random sampling and Permutation\n",
    "#### If you want to draw a random sample from a large dataset for an application or simulation.\n",
    "#### There are a number of ways to do it, one of which is to use 'sample' method for Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hearts, Spades, Clubs, Diamonds\n",
    "suits = ['H', 'S', 'C', 'D']\n",
    "card_val = (list(range(1, 11)) + [10] * 3) * 4\n",
    "base_names = ['A'] + list(range(2, 11)) + ['J', 'K', 'Q']\n",
    "cards = []\n",
    "\n",
    "for suit in suits:\n",
    "    cards.extend(str(num) + suit for num in base_names)\n",
    "    \n",
    "deck = pd.Series(card_val, index=cards)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We have created a Series whose index contains card names and values used in Blackjack and other games. We have kept ace as 1 for simplicity.\n",
    "#### We can draw a hand of any number of cards from the deck using 'sample'.\n",
    "#### We can also get random cards from each suit. We have the suit as the last character of each card, which we can use to group and then use apply for the sample function.\n",
    "#### To not show the suit, we can pass 'group_keys=False' with the groupby function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AH      1\n",
       "2H      2\n",
       "3H      3\n",
       "4H      4\n",
       "5H      5\n",
       "6H      6\n",
       "7H      7\n",
       "8H      8\n",
       "9H      9\n",
       "10H    10\n",
       "JH     10\n",
       "KH     10\n",
       "QH     10\n",
       "dtype: int64"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deck[:13]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9S     9\n",
       "8D     8\n",
       "QS    10\n",
       "7D     7\n",
       "5S     5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def draw(deck, n=5):\n",
    "    return deck.sample(n)\n",
    "\n",
    "draw(deck)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "C  4C      4\n",
       "   KC     10\n",
       "D  JD     10\n",
       "   7D      7\n",
       "H  7H      7\n",
       "   QH     10\n",
       "S  9S      9\n",
       "   10S    10\n",
       "dtype: int64"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get last letter for suit\n",
    "get_suit = lambda card: card[-1]\n",
    "\n",
    "deck.groupby(get_suit).apply(draw, n=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AC     1\n",
       "3C     3\n",
       "3D     3\n",
       "KD    10\n",
       "JH    10\n",
       "3H     3\n",
       "JS    10\n",
       "9S     9\n",
       "dtype: int64"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deck.groupby(get_suit, group_keys=False).apply(draw, n=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Group WeightedAverage and Correlation\n",
    "#### Using the split-apply-combine paradigm, groupby operates between columns in a DataFrame or two Series. this makes group aggregation methods possible.    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>data</th>\n",
       "      <th>weights</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>1.638956</td>\n",
       "      <td>0.548527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a</td>\n",
       "      <td>-0.516697</td>\n",
       "      <td>0.662717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a</td>\n",
       "      <td>0.639473</td>\n",
       "      <td>-1.285644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a</td>\n",
       "      <td>-1.341274</td>\n",
       "      <td>1.839168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b</td>\n",
       "      <td>0.656749</td>\n",
       "      <td>-0.664400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>b</td>\n",
       "      <td>1.108143</td>\n",
       "      <td>0.507994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>b</td>\n",
       "      <td>-0.146337</td>\n",
       "      <td>-0.402249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>b</td>\n",
       "      <td>2.183043</td>\n",
       "      <td>-0.594460</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  category      data   weights\n",
       "0        a  1.638956  0.548527\n",
       "1        a -0.516697  0.662717\n",
       "2        a  0.639473 -1.285644\n",
       "3        a -1.341274  1.839168\n",
       "4        b  0.656749 -0.664400\n",
       "5        b  1.108143  0.507994\n",
       "6        b -0.146337 -0.402249\n",
       "7        b  2.183043 -0.594460"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'category': ['a', 'a', 'a', 'a',\n",
    "                               'b', 'b', 'b', 'b'],\n",
    "                  'data': np.random.randn(8),\n",
    "                  'weights': np.random.randn(8)})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "category\n",
       "a   -1.548291\n",
       "b    0.964589\n",
       "dtype: float64"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped = df.groupby('category')\n",
    "get_wavg = lambda g: np.average(g['data'], weights=g['weights'])\n",
    "\n",
    "grouped.apply(get_wavg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's take an example of financial dataset taken from Yahoo! Finance with end-of-day prices with S&P index (SPX) of few stocks.\n",
    "#### A task of interest is getting yearly correlation of daily return with SPX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "DatetimeIndex: 2214 entries, 2003-01-02 to 2011-10-14\n",
      "Data columns (total 4 columns):\n",
      "AAPL    2214 non-null float64\n",
      "MSFT    2214 non-null float64\n",
      "XOM     2214 non-null float64\n",
      "SPX     2214 non-null float64\n",
      "dtypes: float64(4)\n",
      "memory usage: 86.5 KB\n"
     ]
    }
   ],
   "source": [
    "close_px = pd.read_csv('examples/stock_px_2.csv', parse_dates = True,\n",
    "                      index_col = 0)\n",
    "\n",
    "close_px.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AAPL</th>\n",
       "      <th>MSFT</th>\n",
       "      <th>XOM</th>\n",
       "      <th>SPX</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2011-10-11</th>\n",
       "      <td>400.29</td>\n",
       "      <td>27.00</td>\n",
       "      <td>76.27</td>\n",
       "      <td>1195.54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011-10-12</th>\n",
       "      <td>402.19</td>\n",
       "      <td>26.96</td>\n",
       "      <td>77.16</td>\n",
       "      <td>1207.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011-10-13</th>\n",
       "      <td>408.43</td>\n",
       "      <td>27.18</td>\n",
       "      <td>76.37</td>\n",
       "      <td>1203.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011-10-14</th>\n",
       "      <td>422.00</td>\n",
       "      <td>27.27</td>\n",
       "      <td>78.11</td>\n",
       "      <td>1224.58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              AAPL   MSFT    XOM      SPX\n",
       "2011-10-11  400.29  27.00  76.27  1195.54\n",
       "2011-10-12  402.19  26.96  77.16  1207.25\n",
       "2011-10-13  408.43  27.18  76.37  1203.66\n",
       "2011-10-14  422.00  27.27  78.11  1224.58"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "close_px[-4:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can start by creating a function computing pairwise correlation of each column with SPX column.\n",
    "#### Next we compute the percent change using 'pct_change'. We then group these changes by year, which can be extracted from each row label with a 1 line function.\n",
    "#### We can also compute inter-column correlations. eg - annual correlations between Apple and Microsoft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AAPL</th>\n",
       "      <th>MSFT</th>\n",
       "      <th>XOM</th>\n",
       "      <th>SPX</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2003</th>\n",
       "      <td>0.541124</td>\n",
       "      <td>0.745174</td>\n",
       "      <td>0.661265</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004</th>\n",
       "      <td>0.374283</td>\n",
       "      <td>0.588531</td>\n",
       "      <td>0.557742</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005</th>\n",
       "      <td>0.467540</td>\n",
       "      <td>0.562374</td>\n",
       "      <td>0.631010</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>0.428267</td>\n",
       "      <td>0.406126</td>\n",
       "      <td>0.518514</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007</th>\n",
       "      <td>0.508118</td>\n",
       "      <td>0.658770</td>\n",
       "      <td>0.786264</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>0.681434</td>\n",
       "      <td>0.804626</td>\n",
       "      <td>0.828303</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>0.707103</td>\n",
       "      <td>0.654902</td>\n",
       "      <td>0.797921</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>0.710105</td>\n",
       "      <td>0.730118</td>\n",
       "      <td>0.839057</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011</th>\n",
       "      <td>0.691931</td>\n",
       "      <td>0.800996</td>\n",
       "      <td>0.859975</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          AAPL      MSFT       XOM  SPX\n",
       "2003  0.541124  0.745174  0.661265  1.0\n",
       "2004  0.374283  0.588531  0.557742  1.0\n",
       "2005  0.467540  0.562374  0.631010  1.0\n",
       "2006  0.428267  0.406126  0.518514  1.0\n",
       "2007  0.508118  0.658770  0.786264  1.0\n",
       "2008  0.681434  0.804626  0.828303  1.0\n",
       "2009  0.707103  0.654902  0.797921  1.0\n",
       "2010  0.710105  0.730118  0.839057  1.0\n",
       "2011  0.691931  0.800996  0.859975  1.0"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spx_corr = lambda x: x.corrwith(x['SPX'])\n",
    "rets = close_px.pct_change().dropna()\n",
    "get_year = lambda x: x.year\n",
    "\n",
    "by_year = rets.groupby(get_year)\n",
    "by_year.apply(spx_corr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2003    0.480868\n",
       "2004    0.259024\n",
       "2005    0.300093\n",
       "2006    0.161735\n",
       "2007    0.417738\n",
       "2008    0.611901\n",
       "2009    0.432738\n",
       "2010    0.571946\n",
       "2011    0.581987\n",
       "dtype: float64"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "by_year.apply(lambda g: g['AAPL'].corr(g['MSFT']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Group-Wise Linear Regression\n",
    "#### groupby can also be used to perform complex group-wise statistical analysis, as long as the function returns pandas object or scalar value.\n",
    "#### eg - We can perform Ordinary Least Square (OLS) regression of chunks of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SPX</th>\n",
       "      <th>intercept</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2003</th>\n",
       "      <td>1.195406</td>\n",
       "      <td>0.000710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004</th>\n",
       "      <td>1.363463</td>\n",
       "      <td>0.004201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005</th>\n",
       "      <td>1.766415</td>\n",
       "      <td>0.003246</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>1.645496</td>\n",
       "      <td>0.000080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007</th>\n",
       "      <td>1.198761</td>\n",
       "      <td>0.003438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>0.968016</td>\n",
       "      <td>-0.001110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>0.879103</td>\n",
       "      <td>0.002954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>1.052608</td>\n",
       "      <td>0.001261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011</th>\n",
       "      <td>0.806605</td>\n",
       "      <td>0.001514</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           SPX  intercept\n",
       "2003  1.195406   0.000710\n",
       "2004  1.363463   0.004201\n",
       "2005  1.766415   0.003246\n",
       "2006  1.645496   0.000080\n",
       "2007  1.198761   0.003438\n",
       "2008  0.968016  -0.001110\n",
       "2009  0.879103   0.002954\n",
       "2010  1.052608   0.001261\n",
       "2011  0.806605   0.001514"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "def regress(data, yvar, xvars):\n",
    "    Y = data[yvar]\n",
    "    X = data[xvars]\n",
    "    X['intercept'] = 1.\n",
    "    result = sm.OLS(Y, X).fit()\n",
    "    return result.params\n",
    "\n",
    "by_year.apply(regress, 'AAPL', ['SPX'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pivot Tables and Cross-Tabulation\n",
    "#### 'pivot table' is a data summarization tool used in spreadsheets and other data analysis software.\n",
    "#### It aggregates table by one or more keys, arranging data in a rectangle with some group keys along rows and others along columns.\n",
    "#### Pivot tables are possible in pandas through 'groupby' combined with 'reshape' utilizing hierarchical indexing.\n",
    "#### DataFrame also has 'pivot_table' method, and there is also a top-level 'pandas.pivot_table' function.\n",
    "#### pivot_table also add partial totals, known as 'margins', in addition to the groupby interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>size</th>\n",
       "      <th>tip</th>\n",
       "      <th>tip_pct</th>\n",
       "      <th>total_bill</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Fri</th>\n",
       "      <th>No</th>\n",
       "      <td>2.250000</td>\n",
       "      <td>2.812500</td>\n",
       "      <td>0.151650</td>\n",
       "      <td>18.420000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>2.066667</td>\n",
       "      <td>2.714000</td>\n",
       "      <td>0.174783</td>\n",
       "      <td>16.813333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sat</th>\n",
       "      <th>No</th>\n",
       "      <td>2.555556</td>\n",
       "      <td>3.102889</td>\n",
       "      <td>0.158048</td>\n",
       "      <td>19.661778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>2.476190</td>\n",
       "      <td>2.875476</td>\n",
       "      <td>0.147906</td>\n",
       "      <td>21.276667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Sun</th>\n",
       "      <th>No</th>\n",
       "      <td>2.929825</td>\n",
       "      <td>3.167895</td>\n",
       "      <td>0.160113</td>\n",
       "      <td>20.506667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>2.578947</td>\n",
       "      <td>3.516842</td>\n",
       "      <td>0.187250</td>\n",
       "      <td>24.120000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Thur</th>\n",
       "      <th>No</th>\n",
       "      <td>2.488889</td>\n",
       "      <td>2.673778</td>\n",
       "      <td>0.160298</td>\n",
       "      <td>17.113111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>2.352941</td>\n",
       "      <td>3.030000</td>\n",
       "      <td>0.163863</td>\n",
       "      <td>19.190588</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 size       tip   tip_pct  total_bill\n",
       "day  smoker                                          \n",
       "Fri  No      2.250000  2.812500  0.151650   18.420000\n",
       "     Yes     2.066667  2.714000  0.174783   16.813333\n",
       "Sat  No      2.555556  3.102889  0.158048   19.661778\n",
       "     Yes     2.476190  2.875476  0.147906   21.276667\n",
       "Sun  No      2.929825  3.167895  0.160113   20.506667\n",
       "     Yes     2.578947  3.516842  0.187250   24.120000\n",
       "Thur No      2.488889  2.673778  0.160298   17.113111\n",
       "     Yes     2.352941  3.030000  0.163863   19.190588"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.pivot_table(index=['day', 'smoker'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### pivot_table allows you to aggregate by certain columns, group by certaing columns with 'index' parameters, and define columns with 'columns' parameter.\n",
    "#### We could augment this table to include partial totals by passing 'margins=True'. It will add All row and column labels with corresponding values being group statistics for all data within single tier.\n",
    "#### The 'All' values are aggregations without taking into account the present groups in the columns or rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">size</th>\n",
       "      <th colspan=\"2\" halign=\"left\">tip_pct</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>smoker</th>\n",
       "      <th>No</th>\n",
       "      <th>Yes</th>\n",
       "      <th>No</th>\n",
       "      <th>Yes</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <th>day</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">Dinner</th>\n",
       "      <th>Fri</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.222222</td>\n",
       "      <td>0.139622</td>\n",
       "      <td>0.165347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sat</th>\n",
       "      <td>2.555556</td>\n",
       "      <td>2.476190</td>\n",
       "      <td>0.158048</td>\n",
       "      <td>0.147906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sun</th>\n",
       "      <td>2.929825</td>\n",
       "      <td>2.578947</td>\n",
       "      <td>0.160113</td>\n",
       "      <td>0.187250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thur</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.159744</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Lunch</th>\n",
       "      <th>Fri</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.833333</td>\n",
       "      <td>0.187735</td>\n",
       "      <td>0.188937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thur</th>\n",
       "      <td>2.500000</td>\n",
       "      <td>2.352941</td>\n",
       "      <td>0.160311</td>\n",
       "      <td>0.163863</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 size             tip_pct          \n",
       "smoker             No       Yes        No       Yes\n",
       "time   day                                         \n",
       "Dinner Fri   2.000000  2.222222  0.139622  0.165347\n",
       "       Sat   2.555556  2.476190  0.158048  0.147906\n",
       "       Sun   2.929825  2.578947  0.160113  0.187250\n",
       "       Thur  2.000000       NaN  0.159744       NaN\n",
       "Lunch  Fri   3.000000  1.833333  0.187735  0.188937\n",
       "       Thur  2.500000  2.352941  0.160311  0.163863"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.pivot_table(['tip_pct', 'size'], index=['time', 'day'],\n",
    "                columns='smoker')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">size</th>\n",
       "      <th colspan=\"3\" halign=\"left\">tip_pct</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>smoker</th>\n",
       "      <th>No</th>\n",
       "      <th>Yes</th>\n",
       "      <th>All</th>\n",
       "      <th>No</th>\n",
       "      <th>Yes</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <th>day</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">Dinner</th>\n",
       "      <th>Fri</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.222222</td>\n",
       "      <td>2.166667</td>\n",
       "      <td>0.139622</td>\n",
       "      <td>0.165347</td>\n",
       "      <td>0.158916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sat</th>\n",
       "      <td>2.555556</td>\n",
       "      <td>2.476190</td>\n",
       "      <td>2.517241</td>\n",
       "      <td>0.158048</td>\n",
       "      <td>0.147906</td>\n",
       "      <td>0.153152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sun</th>\n",
       "      <td>2.929825</td>\n",
       "      <td>2.578947</td>\n",
       "      <td>2.842105</td>\n",
       "      <td>0.160113</td>\n",
       "      <td>0.187250</td>\n",
       "      <td>0.166897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thur</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.159744</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.159744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Lunch</th>\n",
       "      <th>Fri</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.833333</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.187735</td>\n",
       "      <td>0.188937</td>\n",
       "      <td>0.188765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thur</th>\n",
       "      <td>2.500000</td>\n",
       "      <td>2.352941</td>\n",
       "      <td>2.459016</td>\n",
       "      <td>0.160311</td>\n",
       "      <td>0.163863</td>\n",
       "      <td>0.161301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <th></th>\n",
       "      <td>2.668874</td>\n",
       "      <td>2.408602</td>\n",
       "      <td>2.569672</td>\n",
       "      <td>0.159328</td>\n",
       "      <td>0.163196</td>\n",
       "      <td>0.160803</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 size                       tip_pct                    \n",
       "smoker             No       Yes       All        No       Yes       All\n",
       "time   day                                                             \n",
       "Dinner Fri   2.000000  2.222222  2.166667  0.139622  0.165347  0.158916\n",
       "       Sat   2.555556  2.476190  2.517241  0.158048  0.147906  0.153152\n",
       "       Sun   2.929825  2.578947  2.842105  0.160113  0.187250  0.166897\n",
       "       Thur  2.000000       NaN  2.000000  0.159744       NaN  0.159744\n",
       "Lunch  Fri   3.000000  1.833333  2.000000  0.187735  0.188937  0.188765\n",
       "       Thur  2.500000  2.352941  2.459016  0.160311  0.163863  0.161301\n",
       "All          2.668874  2.408602  2.569672  0.159328  0.163196  0.160803"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.pivot_table(['tip_pct', 'size'], index=['time', 'day'],\n",
    "                columns='smoker', margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To use a different aggregation function, pass it to 'aggfunc'. eg - 'count' or len will give cross-tabulation (frequencies) of group sizes.\n",
    "#### If some combinations are empty (NA), you can pass a 'fill_value'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>Fri</th>\n",
       "      <th>Sat</th>\n",
       "      <th>Sun</th>\n",
       "      <th>Thur</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Dinner</th>\n",
       "      <th>No</th>\n",
       "      <td>3.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>106.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>9.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>70.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Lunch</th>\n",
       "      <th>No</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>44.0</td>\n",
       "      <td>45.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17.0</td>\n",
       "      <td>23.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <th></th>\n",
       "      <td>19.0</td>\n",
       "      <td>87.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>244.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "day             Fri   Sat   Sun  Thur    All\n",
       "time   smoker                               \n",
       "Dinner No       3.0  45.0  57.0   1.0  106.0\n",
       "       Yes      9.0  42.0  19.0   NaN   70.0\n",
       "Lunch  No       1.0   NaN   NaN  44.0   45.0\n",
       "       Yes      6.0   NaN   NaN  17.0   23.0\n",
       "All            19.0  87.0  76.0  62.0  244.0"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.pivot_table('tip_pct', index=['time', 'smoker'], columns='day', \n",
    "                 aggfunc=len, margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>Fri</th>\n",
       "      <th>Sat</th>\n",
       "      <th>Sun</th>\n",
       "      <th>Thur</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>smoker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"11\" valign=\"top\">Dinner</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.137931</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.325733</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>No</th>\n",
       "      <td>0.139622</td>\n",
       "      <td>0.162705</td>\n",
       "      <td>0.168859</td>\n",
       "      <td>0.159744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.171297</td>\n",
       "      <td>0.148668</td>\n",
       "      <td>0.207893</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.154661</td>\n",
       "      <td>0.152663</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.144995</td>\n",
       "      <td>0.152660</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">4</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.150096</td>\n",
       "      <td>0.148143</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.117750</td>\n",
       "      <td>0.124515</td>\n",
       "      <td>0.193370</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">5</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.206928</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.106572</td>\n",
       "      <td>0.065660</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.103799</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">Lunch</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.181728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.223776</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.166005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.181969</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.158843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>No</th>\n",
       "      <td>0.187735</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.084246</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.204952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">4</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.138919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.155410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.121389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <th>No</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.173706</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "day                      Fri       Sat       Sun      Thur\n",
       "time   size smoker                                        \n",
       "Dinner 1    No      0.000000  0.137931  0.000000  0.000000\n",
       "            Yes     0.000000  0.325733  0.000000  0.000000\n",
       "       2    No      0.139622  0.162705  0.168859  0.159744\n",
       "            Yes     0.171297  0.148668  0.207893  0.000000\n",
       "       3    No      0.000000  0.154661  0.152663  0.000000\n",
       "            Yes     0.000000  0.144995  0.152660  0.000000\n",
       "       4    No      0.000000  0.150096  0.148143  0.000000\n",
       "            Yes     0.117750  0.124515  0.193370  0.000000\n",
       "       5    No      0.000000  0.000000  0.206928  0.000000\n",
       "            Yes     0.000000  0.106572  0.065660  0.000000\n",
       "       6    No      0.000000  0.000000  0.103799  0.000000\n",
       "Lunch  1    No      0.000000  0.000000  0.000000  0.181728\n",
       "            Yes     0.223776  0.000000  0.000000  0.000000\n",
       "       2    No      0.000000  0.000000  0.000000  0.166005\n",
       "            Yes     0.181969  0.000000  0.000000  0.158843\n",
       "       3    No      0.187735  0.000000  0.000000  0.084246\n",
       "            Yes     0.000000  0.000000  0.000000  0.204952\n",
       "       4    No      0.000000  0.000000  0.000000  0.138919\n",
       "            Yes     0.000000  0.000000  0.000000  0.155410\n",
       "       5    No      0.000000  0.000000  0.000000  0.121389\n",
       "       6    No      0.000000  0.000000  0.000000  0.173706"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.pivot_table('tip_pct', index=['time', 'size', 'smoker'], columns='day', \n",
    "                 aggfunc='mean', fill_value=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-Tabulations: Crosstab\n",
    "#### Cross-Tabulation (crosstab for short) is a special case of pivot table that computes group frequencies.\n",
    "#### In some scenarios, 'pandas.crosstab' is a more convenient way to summarize data.\n",
    "#### The first 2 arguments can be either an array or Series or list of arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sample</th>\n",
       "      <th>Nationality</th>\n",
       "      <th>Handedness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>USA</td>\n",
       "      <td>Right</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>JP</td>\n",
       "      <td>Left</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>USA</td>\n",
       "      <td>Left</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>JP</td>\n",
       "      <td>Right</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>JP</td>\n",
       "      <td>Left</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>USA</td>\n",
       "      <td>Left</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sample Nationality Handedness\n",
       "0       1         USA      Right\n",
       "1       2          JP       Left\n",
       "2       3         USA       Left\n",
       "3       4          JP      Right\n",
       "4       5          JP       Left\n",
       "5       6         USA       Left"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = {0: [1, 'USA', 'Right'],\n",
    "    1: [2, 'JP', 'Left'],\n",
    "    2: [3, 'USA', 'Left'],\n",
    "    3: [4, 'JP', 'Right'],\n",
    "    4: [5, 'JP', 'Left'],\n",
    "    5: [6, 'USA', 'Left']}\n",
    "data = pd.DataFrame.from_dict(d, orient='index',\n",
    "                              columns=['Sample', 'Nationality', 'Handedness'])\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Handedness</th>\n",
       "      <th>Left</th>\n",
       "      <th>Right</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nationality</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>JP</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>USA</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Handedness   Left  Right  All\n",
       "Nationality                  \n",
       "JP              2      1    3\n",
       "USA             2      1    3\n",
       "All             4      2    6"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(data.Nationality, data.Handedness, margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smoker</th>\n",
       "      <th>No</th>\n",
       "      <th>Yes</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <th>day</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">Dinner</th>\n",
       "      <th>Fri</th>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sat</th>\n",
       "      <td>45</td>\n",
       "      <td>42</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sun</th>\n",
       "      <td>57</td>\n",
       "      <td>19</td>\n",
       "      <td>76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thur</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Lunch</th>\n",
       "      <th>Fri</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thur</th>\n",
       "      <td>44</td>\n",
       "      <td>17</td>\n",
       "      <td>61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <th></th>\n",
       "      <td>151</td>\n",
       "      <td>93</td>\n",
       "      <td>244</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "smoker        No  Yes  All\n",
       "time   day                \n",
       "Dinner Fri     3    9   12\n",
       "       Sat    45   42   87\n",
       "       Sun    57   19   76\n",
       "       Thur    1    0    1\n",
       "Lunch  Fri     1    6    7\n",
       "       Thur   44   17   61\n",
       "All          151   93  244"
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab([tips.time, tips.day], tips.smoker, margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}