15_matplotlib.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting data with matplotlib\n",
    "\n",
    "Plotting of data is pandas is handled by an external Python module called *matplotlib*. Like pandas it is a large library and has a venerable history (first released in 2003) and so we couldn't hope to cover all its functionality in this course. To see the wide range of possibilities you have with matplotlib see its [example gallery](http://matplotlib.org/gallery.html).\n",
    "\n",
    "Here we will cover the basic uses of it and how it integrates with pandas. While working through these examples you will likely find it very useful to refer to the [matplotlib documentation](http://matplotlib.org/api/pyplot_api.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we import pandas and numpy in the same way as we did previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pandas import Series, DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some matplotlib functionality is provided directly through pandas (such as the `plot()` method as we will see) but for much of it you need to import the matplotlib interface itself.\n",
    "\n",
    "The most common interface to matplotlib is its `pyplot` module which provides a way to affect the current state of matplotlib directly. [By convention](http://matplotlib.org/faq/usage_faq.html#coding-styles) this is imported as `plt`.\n",
    "\n",
    "We also set the figure format to be SVG so that the plots look a little nicer in our Jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%config InlineBackend.figure_format = 'svg'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have imported matplotlib we can start calling its functions. Any functions called on the `plt` object will affect all of matplotlib from that point on in the script."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first need to import some data to plot. Let's start with the data from the pandas section (available from [cetml1659on.dat](https://raw.githubusercontent.com/milliams/data_analysis_python/master/cetml1659on.dat)) and import it into a `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\n",
    "    'data/cetml1659on.dat',  # file name\n",
    "    skiprows=6,  # skip header\n",
    "    sep='\\s+',  # whitespace separated\n",
    "    na_values=['-99.9', '-99.99'],  # NaNs\n",
    ")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas integrates matplotlib directly into itself so any dataframe can be plotted easily simply by calling the `plot()` method on one of the columns. This creates a plot object which you can then edit and alter, for example by setting the axis labels using the `plt.ylabel()` function before displaying it with `plt.show()`.\n",
    "\n",
    "Matplotlib operates on a single global state and calling any function on `plt` will alter that state. Calling `df.plot()` sets the *currently operating* plot. `plt.ylabel()` then alters that state and `plt.show()` displays it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['JAN'].plot()\n",
    "\n",
    "plt.ylabel(r'Temperature ($^\\circ$C)')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "\n",
    "- Make sure you can reproduce the plot above. Try tweaking the labels or which column is plotted.\n",
    "- Try putting in two `plot()` calls with different months (January and July for example) before calling `show()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making it prettier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While it's useful to be able to quickly plot any data we have in front of us, matplotlib's power comes from its configurability. Let's experiment with a dataset and see how much we can change the plot.\n",
    "\n",
    "We'll start with a simple `DataFrame` containing two columns, one with the values of a cosine, the other with the values of a sine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.linspace(-np.pi, np.pi, 256, endpoint=True)\n",
    "data = {'cos': np.cos(X), 'sin': np.sin(X)}\n",
    "trig = DataFrame(index=X, data=data)\n",
    "\n",
    "trig.plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that it has plotted the sine and cosine curves between $\\pi$ and $-\\pi$. Now, let's go through and see how we can affect the display of this plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing colours and line widths\n",
    "\n",
    "First step, we want to have the cosine in blue and the sine in red and a slighty thicker line for both of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trig.cos.plot(color=\"blue\", linewidth=2.5, linestyle=\"-\")\n",
    "trig.sin.plot(color=\"red\", linewidth=2.5, linestyle=\"-\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "\n",
    "- Using the temperature dataset, set the colours of the July and January lines to a warm colour and a cool colour.\n",
    "- Add in the yearly average column to the plot with a dashed line style."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting limits\n",
    "\n",
    "Current limits of the figure are a bit too tight and we want to make some space in order to clearly see all data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trig.cos.plot(color=\"blue\", linewidth=2.5, linestyle=\"-\")\n",
    "trig.sin.plot(color=\"red\", linewidth=2.5, linestyle=\"-\")\n",
    "\n",
    "### New code\n",
    "plt.xlim(trig.index.min() * 1.1, trig.index.max() * 1.1)\n",
    "plt.ylim(trig.cos.min() * 1.1, trig.cos.max() * 1.1)\n",
    "### End of new code\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting ticks\n",
    "\n",
    "Current ticks are not ideal because they do not show the interesting values ($\\pm\\pi$,$\\pm\\frac{\\pi}{2}$) for sine and cosine. We’ll change them such that they show only these values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trig.cos.plot(color=\"blue\", linewidth=2.5, linestyle=\"-\")\n",
    "trig.sin.plot(color=\"red\", linewidth=2.5, linestyle=\"-\")\n",
    "\n",
    "plt.xlim(trig.index.min() * 1.1, trig.index.max() * 1.1)\n",
    "plt.ylim(trig.cos.min() * 1.1, trig.cos.max() * 1.1)\n",
    "\n",
    "### New code\n",
    "plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi])\n",
    "plt.yticks([-1, 0, +1])\n",
    "### End of new code\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting tick labels\n",
    "\n",
    "Ticks are now properly placed but their label is not very explicit. We could guess that 3.142 is $\\pi$ but it would be better to make it explicit. When we set tick values, we can also provide a corresponding label in the second argument list. Note that we’ll use LaTeX to allow for nice rendering of the label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trig.cos.plot(color=\"blue\", linewidth=2.5, linestyle=\"-\")\n",
    "trig.sin.plot(color=\"red\", linewidth=2.5, linestyle=\"-\")\n",
    "\n",
    "plt.xlim(trig.index.min() * 1.1, trig.index.max() * 1.1)\n",
    "plt.ylim(trig.cos.min() * 1.1, trig.cos.max() * 1.1)\n",
    "\n",
    "### New code\n",
    "plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi],\n",
    "           [r'$-\\pi$', r'$-\\pi/2$', r'$0$', r'$+\\pi/2$', r'$+\\pi$'])\n",
    "\n",
    "plt.yticks([-1, 0, +1],\n",
    "           [r'$-1$', r'$0$', r'$+1$'])\n",
    "### End of new code\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Moving spines\n",
    "\n",
    "Spines are the lines connecting the axis tick marks and noting the boundaries of the data area. They can be placed at arbitrary positions and until now, they were on the border of the axis. We’ll change that since we want to have them in the middle. Since there are four of them (top/bottom/left/right), we’ll discard the top and right by setting their color to none and we’ll move the bottom and left ones to coordinate 0 in data space coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trig.cos.plot(color=\"blue\", linewidth=2.5, linestyle=\"-\")\n",
    "trig.sin.plot(color=\"red\", linewidth=2.5, linestyle=\"-\")\n",
    "\n",
    "plt.xlim(trig.index.min() * 1.1, trig.index.max() * 1.1)\n",
    "plt.ylim(trig.cos.min() * 1.1, trig.cos.max() * 1.1)\n",
    "\n",
    "plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi])\n",
    "plt.yticks([-1, 0, +1])\n",
    "\n",
    "plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi],\n",
    "           [r'$-\\pi$', r'$-\\pi/2$', r'$0$', r'$+\\pi/2$', r'$+\\pi$'])\n",
    "\n",
    "plt.yticks([-1, 0, +1],\n",
    "           [r'$-1$', r'$0$', r'$+1$'])\n",
    "\n",
    "### New code\n",
    "ax = plt.gca()  # gca stands for 'get current axis'\n",
    "ax.spines['right'].set_color('none')\n",
    "ax.spines['top'].set_color('none')\n",
    "ax.xaxis.set_ticks_position('bottom')\n",
    "ax.spines['bottom'].set_position(('data',0))\n",
    "ax.yaxis.set_ticks_position('left')\n",
    "ax.spines['left'].set_position(('data',0))\n",
    "### End of new code\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a legend\n",
    "\n",
    "Let’s add a legend in the upper left corner. This only requires adding the keyword argument label (that will be used in the legend box) to the plot commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "trig.cos.plot(color=\"blue\", linewidth=2.5, linestyle=\"-\")\n",
    "trig.sin.plot(color=\"red\", linewidth=2.5, linestyle=\"-\")\n",
    "\n",
    "plt.xlim(trig.index.min() * 1.1, trig.index.max() * 1.1)\n",
    "plt.ylim(trig.cos.min() * 1.1, trig.cos.max() * 1.1)\n",
    "\n",
    "plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi])\n",
    "plt.yticks([-1, 0, +1])\n",
    "\n",
    "plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi],\n",
    "           [r'$-\\pi$', r'$-\\pi/2$', r'$0$', r'$+\\pi/2$', r'$+\\pi$'])\n",
    "\n",
    "plt.yticks([-1, 0, +1],\n",
    "           [r'$-1$', r'$0$', r'$+1$'])\n",
    "\n",
    "ax = plt.gca()  # gca stands for 'get current axis'\n",
    "ax.spines['right'].set_color('none')\n",
    "ax.spines['top'].set_color('none')\n",
    "ax.xaxis.set_ticks_position('bottom')\n",
    "ax.spines['bottom'].set_position(('data',0))\n",
    "ax.yaxis.set_ticks_position('left')\n",
    "ax.spines['left'].set_position(('data',0))\n",
    "\n",
    "### New code\n",
    "plt.legend(loc='upper left')\n",
    "### End of new code\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Annotate some points\n",
    "\n",
    "Let’s annotate some interesting points using the annotate command. We chose the $\\frac{2}{3}\\pi$ value and we want to annotate both the sine and the cosine. We’ll first draw a marker on the curve as well as a straight dotted line. Then, we’ll use the annotate command to display some text with an arrow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trig.cos.plot(color=\"blue\", linewidth=2.5, linestyle=\"-\")\n",
    "trig.sin.plot(color=\"red\", linewidth=2.5, linestyle=\"-\")\n",
    "\n",
    "plt.xlim(trig.index.min() * 1.1, trig.index.max() * 1.1)\n",
    "plt.ylim(trig.cos.min() * 1.1, trig.cos.max() * 1.1)\n",
    "\n",
    "plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi])\n",
    "plt.yticks([-1, 0, +1])\n",
    "\n",
    "plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi],\n",
    "           [r'$-\\pi$', r'$-\\pi/2$', r'$0$', r'$+\\pi/2$', r'$+\\pi$'])\n",
    "\n",
    "plt.yticks([-1, 0, +1],\n",
    "           [r'$-1$', r'$0$', r'$+1$'])\n",
    "\n",
    "ax = plt.gca()  # gca stands for 'get current axis'\n",
    "ax.spines['right'].set_color('none')\n",
    "ax.spines['top'].set_color('none')\n",
    "ax.xaxis.set_ticks_position('bottom')\n",
    "ax.spines['bottom'].set_position(('data',0))\n",
    "ax.yaxis.set_ticks_position('left')\n",
    "ax.spines['left'].set_position(('data',0))\n",
    "\n",
    "plt.legend(loc='upper left')\n",
    "\n",
    "### New code\n",
    "t = 2 * np.pi / 3\n",
    "plt.plot([t, t], [0, np.cos(t)], color='blue', linewidth=2.5, linestyle=\"--\")\n",
    "plt.scatter([t, ], [np.cos(t), ], 50, color='blue')\n",
    "\n",
    "plt.annotate(r'$cos(\\frac{2\\pi}{3})=-\\frac{1}{2}$',\n",
    "             xy=(t, np.cos(t)), xycoords='data',\n",
    "             xytext=(-90, -50), textcoords='offset points', fontsize=16,\n",
    "             arrowprops=dict(arrowstyle=\"->\", connectionstyle=\"arc3,rad=.2\"))\n",
    "\n",
    "plt.plot([t, t],[0, np.sin(t)], color='red', linewidth=2.5, linestyle=\"--\")\n",
    "plt.scatter([t, ],[np.sin(t), ], 50, color='red')\n",
    "\n",
    "plt.annotate(r'$sin(\\frac{2\\pi}{3})=\\frac{\\sqrt{3}}{2}$',\n",
    "             xy=(t, np.sin(t)), xycoords='data',\n",
    "             xytext=(+10, +30), textcoords='offset points', fontsize=16,\n",
    "             arrowprops=dict(arrowstyle=\"->\", connectionstyle=\"arc3,rad=.2\"))\n",
    "### End of new code\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you know how to make different modifications to your plots we can make some of these changes to our temerature data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving plot to a file\n",
    "\n",
    "You can take any plot you've created within Jupyter and save it to a file on disk using the `plt.savefig()` function. You give the function the name of the file to create and it will use whatever format is specified by the name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trig.plot()\n",
    "\n",
    "plt.show()\n",
    "\n",
    "plt.savefig('my_fig.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3\n",
    "\n",
    "- Add in a legend for the data.\n",
    "- Add an annotation to one of the spikes in the data. Make sure the label is placed nicely.\n",
    " - Tip: you can get the year and temperature for a spike using:\n",
    " ```python\n",
    "  warm_winter_year = df['JAN'].idxmax()\n",
    "  warm_winter_temp = df['JAN'].max()\n",
    " ```\n",
    "- Save the figure to a file and display it in your Jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bar charts\n",
    "\n",
    "Of course, Matplotlib can plot more than just line graphs. One of the other most common plot types is a bar chart. Let's work towards plotting a bar chart of the average temperature per decade.\n",
    "\n",
    "Let's start by adding a new column to the data frame which represents the decade. We create it by taking the index (which is a list of years), converting each element to a string and then replacing the fourth character with a `'0'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "years = Series(df.index, index=df.index).apply(str)\n",
    "decade = years.apply(lambda x: x[:3]+'0')\n",
    "\n",
    "df['decade'] = decade\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have our decade column, we can use Pandas `groupby()` function to gather our data by decade and then aggregate it by taking the mean of each decade."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "by_decade = df.groupby('decade')\n",
    "agg = by_decade.aggregate(np.mean)\n",
    "\n",
    "agg.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, `agg` is a standard Pandas `DataFrame` so we can plot it like any other, by putting `.bar` after the `plot` call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agg.YEAR.plot.bar()\n",
    "\n",
    "plt.ylabel(r'Temperature ($^\\circ$C)')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4\n",
    "\n",
    "1. Plot a bar chart of the average temperature per *century*.\n",
    " - Set the limits of the y-axis to zoom in on the data.\n",
    "\n",
    "2. Plot a *histogram* of the average annual temperature\n",
    " - Make sure that the x-axis is labelled correctly.\n",
    " - Tip: Look in the [documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html) for the right command to run\n",
    " \n",
    "3. Plot a scatter plot of each year's February temperature plotted against that year's January temperature. Is there an obvious correlation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15+"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}