notebooks/dynamic_memory_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Memory Representation in Dialogue Systems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following notebook is the result of an NLP project that explores the question, \"How could interaction be stored in memory, and how can that information be leveraged for further use?\" \n",
    "\n",
    "[Dialog(ue) systems](https://en.wikipedia.org/wiki/Dialog_system) can be quite useful, but have difficulty keeping track of concepts and entities dynamically. Commercial implementations among the likes of Siri, Google Assistant, and Alexa are great for performing simple tasks, but fall short when remembering ad-hoc relationships that regularly present themselves in conversation. For more information on dialogue systems, graph databases, and ontologies as they relate to this project, see the white paper entitled [IPA_Memory](/files/docs/IPA_Memory_Dan_Kondratyuk_2017.04.30.pdf) under the `docs` directory of this repository.\n",
    "\n",
    "To enhance the capabilities of dialogue systems, this notebook will provide a simple software implementation of a model that is intended to by dynamic, incremental, flexible, and interpretable. By forming high-level concepts that evolve over time, this model will evaluate the dialogue system's ability to understand user input. This notebook will show how such a system can update its internal state based on natural language facts, and retrieve results based on natural language questions. See the white paper for more details on the rationale behind these design decisions.\n",
    "\n",
    "The code below is written in Python, and uses a [Neo4j Graph Database](https://neo4j.com/product/) to provide non-volatile storage and efficient querying capabilities.\n",
    "\n",
    "The test corpus is supplied by the [bAbI Tasks Data 1-20 (v1.2)](https://research.fb.com/downloads/babi/). It contains sequences of English sentences to provide the system knowledge of a simple domain involving characters moving to different rooms and interacting with objects. Questions are inserted periodically to evaluate that the system is keeping track of these relationships accurately.\n",
    "\n",
    "## Prerequisites to Running this Notebook\n",
    "- [Python](https://www.python.org/downloads/) (3.5+)\n",
    "- Python packages (install via pip): `pandas`, `numpy`, `nltk`, `scikit-learn`, `neo4j-driver`\n",
    "- [Neo4j](https://neo4j.com/download/) (3.1+)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: bAbI QA 1\n",
    "\n",
    "## Process the Text\n",
    "\n",
    "### Import DataFrames\n",
    "First we will use `pandas` to import `qa1_single-supporting-fact_train.txt` from our corpus into a DataFrame called `data`. Every line in this document represents one sentence, which will be split using `nltk`'s word tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Import the necessary packages\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download the all-nltk packages (or all packages)\n",
    "# An OS window should pop up for you to download the appropriate packages\n",
    "nltk.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Read the bAbI data as CSV\n",
    "filename = 'resources/qa1_single-supporting-fact_train.txt'\n",
    "data_qa1 = pd.read_csv(filename, delimiter='\\t', names=['sentence', 'answer', 'factid'])\n",
    "data_qa1 = data_qa1.fillna('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is what the input data looks like. Every `sentence` in this frame can either be a factual statement, or a question about the preceeding statements. Each statement describes four characters moving between six different rooms. The questions periodically ask the room in which a person is currently in, and the objective is to answer them all correctly, matching the corresponding `answer` column (it is blank if the sentence is a statement). The `factid` column indicates the index of the supporting facts for each answer, but we won't be needing it.\n",
    "\n",
    "Due to the nature of the model, training will not be necessary to answer each question. Therefore, the entire document will be used for test evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>answer</th>\n",
       "      <th>factid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1 Mary moved to the bathroom.</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2 John went to the hallway.</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3 Where is Mary?</td>\n",
       "      <td>bathroom</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4 Daniel went back to the hallway.</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5 Sandra moved to the garden.</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6 Where is Daniel?</td>\n",
       "      <td>hallway</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             sentence    answer factid\n",
       "0       1 Mary moved to the bathroom.                 \n",
       "1         2 John went to the hallway.                 \n",
       "2                   3 Where is Mary?   bathroom      1\n",
       "3  4 Daniel went back to the hallway.                 \n",
       "4       5 Sandra moved to the garden.                 \n",
       "5                 6 Where is Daniel?    hallway      4"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_qa1[:6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we process this data frame by splitting the sentences and tagging each sentence by its type (statement or question). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Tag each sentence as a statement (S) or question (Q)\n",
    "tag_sentence = lambda row: 'S' if row.answer == '' else 'Q'\n",
    "data_qa1['type'] = data_qa1.apply(tag_sentence, axis=1)\n",
    "\n",
    "# Use NLTK to tokenize the sentences into arrays of words\n",
    "# If you get an error here, make sure you donwloaded the NLTK packages above\n",
    "tokenize = lambda row: nltk.word_tokenize(row.sentence)[1:]\n",
    "data_qa1.sentence = data_qa1.apply(tokenize, axis=1)\n",
    "\n",
    "# Drop the factid column, as we won't need it\n",
    "data_qa1 = data_qa1.drop('factid', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>answer</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Mary, moved, to, the, bathroom, .]</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[John, went, to, the, hallway, .]</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[Where, is, Mary, ?]</td>\n",
       "      <td>bathroom</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Daniel, went, back, to, the, hallway, .]</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[Sandra, moved, to, the, garden, .]</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[Where, is, Daniel, ?]</td>\n",
       "      <td>hallway</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    sentence    answer type\n",
       "0        [Mary, moved, to, the, bathroom, .]              S\n",
       "1          [John, went, to, the, hallway, .]              S\n",
       "2                       [Where, is, Mary, ?]  bathroom    Q\n",
       "3  [Daniel, went, back, to, the, hallway, .]              S\n",
       "4        [Sandra, moved, to, the, garden, .]              S\n",
       "5                     [Where, is, Daniel, ?]   hallway    Q"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_qa1[:6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We further split the `data_qa1` DataFrame into `statements` and `questions` DataFrames for easy access to all statements and questions respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a DataFrame with just the statements\n",
    "def statements(df):\n",
    "    return df[df.type == 'S'] \\\n",
    "        .reset_index(drop=True) \\\n",
    "        .drop('answer', axis=1) \\\n",
    "        .drop('type', axis=1)\n",
    "\n",
    "# Create a DataFrame with just the questions\n",
    "def questions(df):\n",
    "    return df[df.type == 'Q'] \\\n",
    "        .reset_index(drop=True) \\\n",
    "        .drop('type', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Mary, moved, to, the, bathroom, .]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[John, went, to, the, hallway, .]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[Daniel, went, back, to, the, hallway, .]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Sandra, moved, to, the, garden, .]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    sentence\n",
       "0        [Mary, moved, to, the, bathroom, .]\n",
       "1          [John, went, to, the, hallway, .]\n",
       "2  [Daniel, went, back, to, the, hallway, .]\n",
       "3        [Sandra, moved, to, the, garden, .]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "statements(data_qa1)[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Where, is, Mary, ?]</td>\n",
       "      <td>bathroom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[Where, is, Daniel, ?]</td>\n",
       "      <td>hallway</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 sentence    answer\n",
       "0    [Where, is, Mary, ?]  bathroom\n",
       "1  [Where, is, Daniel, ?]   hallway"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questions(data_qa1)[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract Entities\n",
    "Next, we will extract the relevant entities from each statement and question so that we can more easily reason with these sentences.\n",
    "\n",
    "#### POS Tagging\n",
    "To process each sentence and produce a useful statement or question object, all that is necessary (for this dataset) is to use a part-of-speech tagger. The generated frame below displays the tagged list of (token, word) pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Tag each token as a part of speech\n",
    "pos_tag = lambda row: nltk.pos_tag(row.sentence)\n",
    "data_qa1['tag'] = data_qa1.apply(pos_tag, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Mary, moved, to, the, bathroom, .]</td>\n",
       "      <td>[(Mary, NNP), (moved, VBD), (to, TO), (the, DT...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[John, went, to, the, hallway, .]</td>\n",
       "      <td>[(John, NNP), (went, VBD), (to, TO), (the, DT)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[Where, is, Mary, ?]</td>\n",
       "      <td>[(Where, WRB), (is, VBZ), (Mary, NNP), (?, .)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Daniel, went, back, to, the, hallway, .]</td>\n",
       "      <td>[(Daniel, NNP), (went, VBD), (back, RB), (to, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[Sandra, moved, to, the, garden, .]</td>\n",
       "      <td>[(Sandra, NNP), (moved, VBD), (to, TO), (the, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    sentence  \\\n",
       "0        [Mary, moved, to, the, bathroom, .]   \n",
       "1          [John, went, to, the, hallway, .]   \n",
       "2                       [Where, is, Mary, ?]   \n",
       "3  [Daniel, went, back, to, the, hallway, .]   \n",
       "4        [Sandra, moved, to, the, garden, .]   \n",
       "\n",
       "                                                 tag  \n",
       "0  [(Mary, NNP), (moved, VBD), (to, TO), (the, DT...  \n",
       "1  [(John, NNP), (went, VBD), (to, TO), (the, DT)...  \n",
       "2     [(Where, WRB), (is, VBZ), (Mary, NNP), (?, .)]  \n",
       "3  [(Daniel, NNP), (went, VBD), (back, RB), (to, ...  \n",
       "4  [(Sandra, NNP), (moved, VBD), (to, TO), (the, ...  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_qa1[['sentence', 'tag']][:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Statements\n",
    "Due to the simplicity of the data, each statement can be thought of as a `(subject, relation, object)` triple. We would like to define a function called `extract_statement`, that when given a sequence of statement tokens, produces this triple. For instance,\n",
    "```\n",
    "extract_statement([Mary, moved, to, the, bathroom, .]) = (Mary, moved, bathroom).\n",
    "```\n",
    "This allows one to construct a graph of relationships between objects, as we will see in the next sections. \n",
    "\n",
    "We can use the POS tags in the sentence to achieve this. If there is a word tagged as a proper noun, it is the subject, if there's a verb, it is the relation, and if there's a simple noun, it is the object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def extract_statement(tags):\n",
    "    '''Extracts a (subject, relation, object) triple from each statement based on the POS tags'''\n",
    "    subject, relation, obj = '', '', ''\n",
    "    for word,tag in tags:\n",
    "        if tag == 'NNP':\n",
    "            subject = word\n",
    "        elif tag == 'VBD' or word == 'journeyed': # TODO: 'journeyed' is tagged improperly\n",
    "            relation = word\n",
    "        if tag == 'NNP' or tag == 'NN':\n",
    "            obj = word\n",
    "    return (subject, relation, obj)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "To test the graph, we would like to define another function `extract_question`, that when given a sequence of quesiton tokens, returns the entity that the question is asking for.\n",
    "```\n",
    "extract_question([Where, is, Mary, ?]) = Mary\n",
    "```\n",
    "\n",
    "The result is the subject we are querying for, whose query should return us a room to answer the question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def extract_question(tags):\n",
    "    '''Extracts the entity under discussion from each question based on the POS tags'''\n",
    "    eud = ''\n",
    "    # This will find the last noun in the sentence\n",
    "    for word,tag in tags:\n",
    "        if tag == 'NNP' or tag == 'NN':\n",
    "            eud = word\n",
    "    return eud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, call the appropriate function to extract the correct info."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def extract(row):\n",
    "    '''Extracts the appropriate data given a processed DataFrame row'''\n",
    "    if row.type == 'S':\n",
    "        return extract_statement(row.tag)\n",
    "    else: \n",
    "        return extract_question(row.tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_qa1['extracted'] = data_qa1.apply(extract, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>extracted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Mary, moved, to, the, bathroom, .]</td>\n",
       "      <td>(Mary, moved, bathroom)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[John, went, to, the, hallway, .]</td>\n",
       "      <td>(John, went, hallway)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[Where, is, Mary, ?]</td>\n",
       "      <td>Mary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Daniel, went, back, to, the, hallway, .]</td>\n",
       "      <td>(Daniel, went, hallway)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[Sandra, moved, to, the, garden, .]</td>\n",
       "      <td>(Sandra, moved, garden)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    sentence                extracted\n",
       "0        [Mary, moved, to, the, bathroom, .]  (Mary, moved, bathroom)\n",
       "1          [John, went, to, the, hallway, .]    (John, went, hallway)\n",
       "2                       [Where, is, Mary, ?]                     Mary\n",
       "3  [Daniel, went, back, to, the, hallway, .]  (Daniel, went, hallway)\n",
       "4        [Sandra, moved, to, the, garden, .]  (Sandra, moved, garden)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_qa1[['sentence', 'extracted']][:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voila, extraction is complete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Debug Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are handy debugging functions that we will use for evaluation.\n",
    "\n",
    "This function finds all statements that refer to a person."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def person_statements(person):\n",
    "    '''Get all statements that refer to the specified person'''\n",
    "    stat = statements(data_qa1)\n",
    "    return stat[stat.extracted.map(lambda t: t[0] == person)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, we can find all statements that refer to Sandra."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>tag</th>\n",
       "      <th>extracted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Sandra, moved, to, the, garden, .]</td>\n",
       "      <td>[(Sandra, NNP), (moved, VBD), (to, TO), (the, ...</td>\n",
       "      <td>(Sandra, moved, garden)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[Sandra, journeyed, to, the, bathroom, .]</td>\n",
       "      <td>[(Sandra, NNP), (journeyed, VBD), (to, TO), (t...</td>\n",
       "      <td>(Sandra, journeyed, bathroom)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[Sandra, travelled, to, the, office, .]</td>\n",
       "      <td>[(Sandra, NNP), (travelled, VBD), (to, TO), (t...</td>\n",
       "      <td>(Sandra, travelled, office)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     sentence  \\\n",
       "3         [Sandra, moved, to, the, garden, .]   \n",
       "5   [Sandra, journeyed, to, the, bathroom, .]   \n",
       "10    [Sandra, travelled, to, the, office, .]   \n",
       "\n",
       "                                                  tag  \\\n",
       "3   [(Sandra, NNP), (moved, VBD), (to, TO), (the, ...   \n",
       "5   [(Sandra, NNP), (journeyed, VBD), (to, TO), (t...   \n",
       "10  [(Sandra, NNP), (travelled, VBD), (to, TO), (t...   \n",
       "\n",
       "                        extracted  \n",
       "3         (Sandra, moved, garden)  \n",
       "5   (Sandra, journeyed, bathroom)  \n",
       "10    (Sandra, travelled, office)  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "person_statements('Sandra')[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function finds the `n` most recent statements that refer to a person."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def person_statements_recent(person, n=5):\n",
    "    '''Get the n most recent statements that refer to the specified person in reverse chronological order'''\n",
    "    return person_statements(person)[-n:].iloc[::-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, we can find the 3 most recent statements Daniel has been referred in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>tag</th>\n",
       "      <th>extracted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>[Daniel, went, to, the, garden, .]</td>\n",
       "      <td>[(Daniel, NNP), (went, VBD), (to, TO), (the, D...</td>\n",
       "      <td>(Daniel, went, garden)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>[Daniel, travelled, to, the, kitchen, .]</td>\n",
       "      <td>[(Daniel, NNP), (travelled, VBD), (to, TO), (t...</td>\n",
       "      <td>(Daniel, travelled, kitchen)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1992</th>\n",
       "      <td>[Daniel, moved, to, the, office, .]</td>\n",
       "      <td>[(Daniel, NNP), (moved, VBD), (to, TO), (the, ...</td>\n",
       "      <td>(Daniel, moved, office)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      sentence  \\\n",
       "1999        [Daniel, went, to, the, garden, .]   \n",
       "1996  [Daniel, travelled, to, the, kitchen, .]   \n",
       "1992       [Daniel, moved, to, the, office, .]   \n",
       "\n",
       "                                                    tag  \\\n",
       "1999  [(Daniel, NNP), (went, VBD), (to, TO), (the, D...   \n",
       "1996  [(Daniel, NNP), (travelled, VBD), (to, TO), (t...   \n",
       "1992  [(Daniel, NNP), (moved, VBD), (to, TO), (the, ...   \n",
       "\n",
       "                         extracted  \n",
       "1999        (Daniel, went, garden)  \n",
       "1996  (Daniel, travelled, kitchen)  \n",
       "1992       (Daniel, moved, office)  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "person_statements_recent('Daniel', n=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Graph\n",
    "Once we have processed the data into triples, we can build graphs from them. Below we have defined a couple functions to reset the database and run queries. We will use Neo4j's driver to accomplish this. Note that if the URL or auth credentials are different, you will need to change them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neo4j.v1 import GraphDatabase, basic_auth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a neo4j session\n",
    "# NOTE: Make sure that URL/credentials are correct and that Neo4j is running\n",
    "driver = GraphDatabase.driver('bolt://localhost:7687', auth=basic_auth('neo4j', 'neo4j'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# WARNING: This function will clear the database when run!\n",
    "def reset_db():\n",
    "    '''Remove all nodes and relationships from the database'''\n",
    "    session = driver.session()\n",
    "    session.run('MATCH (n) DETACH DELETE n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create(query, n=0):\n",
    "    '''Given a query, create a graph based on each triple in the extracted statements'''\n",
    "    session = driver.session()\n",
    "    stat = statements(data_qa1)\n",
    "    n = len(stat) if n <= 0 else n\n",
    "    for subject,relation,obj in stat[:n].extracted:\n",
    "        session.run(query, subject=subject, relation=relation, obj=obj)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### V1: Direct relationships\n",
    "One of the first impulses when building the graph is to represent the subject and object as nodes, and the relations as edges between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "reset_db()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a direct relationship between subject and object\n",
    "v1_query = '''\n",
    "    MERGE (s:SUBJECT {name: $subject}) \n",
    "    MERGE (o:OBJECT  {name: $obj}) \n",
    "    MERGE (s)-[r:RELATION {name: $relation}]->(o)\n",
    "'''\n",
    "\n",
    "create(v1_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the query below and see what the graph looks like. Pop open a new tab in the Neo4j browser (default http://localhost:7474/browser/) and run the query:\n",
    "```\n",
    "MATCH (n) RETURN n LIMIT 50\n",
    "```\n",
    "The graph is a reasonable first start, as the relations point each person to where they have been. But this poses a potential problem: how do we know where each person is right now, or where they have been previously? All we can know from the graph is which rooms a person has been in, because they may have visited them all multiple times."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"screenshots/simple-relation.png\",width=700>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### V2: Nodes for relationships\n",
    "One approach is to form a linked list of \"events\". Each event corresponds to a person updating the room that they are in. Since we chose edges to be our relations, we cannot form edges between relations. To alleviate this, we can transform the relation to a node, and draw two edges to form a 3-node triple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reset_db()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Represent each relation as a node\n",
    "v2_query = '''\n",
    "    MERGE (s:SUBJECT {name: $subject})\n",
    "    MERGE (o:OBJECT  {name: $obj})\n",
    "    CREATE (s)-[:R0]->(r:RELATION {name: $relation})-[:R1]->(o)\n",
    "'''\n",
    "\n",
    "create(v2_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the query again and see what changed. This is better, since we can see how often a room has been visited, but still doesn't solve the question as to which room a person is in at any given time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### V3: Linked list of relationships\n",
    "The final step is to build the linked list based on the order in which the relations were created. This will allow us to not only find the room a person is in right now, but produce a list of rooms that they were in, in the order that they were visited."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "reset_db()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Represent each relation as a node, ordered by a linked list (per subject)\n",
    "v3_query = '''\n",
    "    MERGE (s:SUBJECT {name: $subject})\n",
    "    MERGE (o:OBJECT  {name: $obj})\n",
    "    \n",
    "    WITH s,o\n",
    "    \n",
    "    // Create an new relation between the subject and object\n",
    "    CREATE (s)-[:R0]->(r:RELATION {name: $relation})-[:R1]->(o)\n",
    "    CREATE (s)-[h:HEAD]->(r) // Make the newly created relation the head of the list\n",
    "    \n",
    "    WITH s,r,o,h\n",
    "    \n",
    "    // Find the previous head of the list (if none exist, this query will terminate here)\n",
    "    MATCH (s)-[h_prev:HEAD]->(r_prev:RELATION)\n",
    "    WHERE h_prev <> h\n",
    "    \n",
    "    // Complete the link, remove the previous head pointer\n",
    "    CREATE (r_prev)-[:NEXT]->(r)\n",
    "    DELETE h_prev\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "session = driver.session()\n",
    "# Create an index for faster access\n",
    "session.run('CREATE INDEX ON :SUBJECT(name)')\n",
    "session.run('CREATE INDEX ON :RELATION(name)')\n",
    "session.run('CREATE INDEX ON :OBJECT(name)')\n",
    "create(v3_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the new graph out and see what changed. It's helpful to change the colors of the nodes and edges to visualize this better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"screenshots/local-list.png\",width=800>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query the Graph\n",
    "Now we can ask the graph useful questions.\n",
    "#### Find the room a person is in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def find_person(person):\n",
    "    '''Find the room a person is currently in'''\n",
    "    query = '''\n",
    "        MATCH (s:SUBJECT {name:$name})-[:HEAD]->(r:RELATION)-->(o:OBJECT)\n",
    "        RETURN s AS subject, r AS relation, o AS obj\n",
    "    '''\n",
    "    return session.run(query, name=person)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the graph-querying function above we can ask, \"Where is Mary?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kitchen\n"
     ]
    }
   ],
   "source": [
    "# Note: If this is run less than a second after creating the knowledge graph, \n",
    "# the Python driver may cause a race condition where the graph \n",
    "# isn't finished updating, which could give you the wrong answer.\n",
    "session = driver.session()\n",
    "record = find_person('Mary').single()\n",
    "print(record['obj'].get('name'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the graph, Mary is in the kitchen. We can verify that this is true with the debug function below, and we can see the corresponding sentence that generated the relationship as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>tag</th>\n",
       "      <th>extracted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1994</th>\n",
       "      <td>[Mary, journeyed, to, the, kitchen, .]</td>\n",
       "      <td>[(Mary, NNP), (journeyed, VBD), (to, TO), (the...</td>\n",
       "      <td>(Mary, journeyed, kitchen)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    sentence  \\\n",
       "1994  [Mary, journeyed, to, the, kitchen, .]   \n",
       "\n",
       "                                                    tag  \\\n",
       "1994  [(Mary, NNP), (journeyed, VBD), (to, TO), (the...   \n",
       "\n",
       "                       extracted  \n",
       "1994  (Mary, journeyed, kitchen)  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "person_statements_recent('Mary', n=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Find the rooms a person has been in (reverse chronological order)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def find_person_history(person, n=100):\n",
    "    '''Find the list of rooms a person was in, ordered by recency'''\n",
    "    length = str(n) if n >= 1 else ''\n",
    "    \n",
    "    query = '''\n",
    "        MATCH (s:SUBJECT {name:$name})-[:HEAD]->(r:RELATION)-->(o:OBJECT)\n",
    "        MATCH (s)-->(r_prev:RELATION)-[k*1..%s]->(r), (r_prev)-->(o_prev:OBJECT)\n",
    "        \n",
    "        WITH size(k) AS dist, r, o, r_prev, o_prev\n",
    "        ORDER BY size(k)\n",
    "        \n",
    "        WITH r, o, r_prev, o_prev\n",
    "        RETURN [r.name] + collect(r_prev.name) AS relation, [o.name] + collect(o_prev.name) AS obj\n",
    "    '''\n",
    "    query = query % length\n",
    "    \n",
    "    session = driver.session()\n",
    "    record = session.run(query, name=person).single()\n",
    "    history = list(zip(record['relation'], record['obj']))[:-1]\n",
    "    \n",
    "    return history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more advanced question that we get for free based on the graph structure is, \"Where has John been recently?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('went', 'bedroom'),\n",
       " ('went', 'garden'),\n",
       " ('went', 'office'),\n",
       " ('journeyed', 'bedroom'),\n",
       " ('travelled', 'hallway')]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "find_person_history('John', n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that John has been to to those places, in that order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>tag</th>\n",
       "      <th>extracted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>[John, went, back, to, the, bedroom, .]</td>\n",
       "      <td>[(John, NNP), (went, VBD), (back, RB), (to, TO...</td>\n",
       "      <td>(John, went, bedroom)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1989</th>\n",
       "      <td>[John, went, back, to, the, garden, .]</td>\n",
       "      <td>[(John, NNP), (went, VBD), (back, RB), (to, TO...</td>\n",
       "      <td>(John, went, garden)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1986</th>\n",
       "      <td>[John, went, back, to, the, office, .]</td>\n",
       "      <td>[(John, NNP), (went, VBD), (back, RB), (to, TO...</td>\n",
       "      <td>(John, went, office)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1982</th>\n",
       "      <td>[John, journeyed, to, the, bedroom, .]</td>\n",
       "      <td>[(John, NNP), (journeyed, NN), (to, TO), (the,...</td>\n",
       "      <td>(John, journeyed, bedroom)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1979</th>\n",
       "      <td>[John, travelled, to, the, hallway, .]</td>\n",
       "      <td>[(John, NNP), (travelled, VBD), (to, TO), (the...</td>\n",
       "      <td>(John, travelled, hallway)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     sentence  \\\n",
       "1995  [John, went, back, to, the, bedroom, .]   \n",
       "1989   [John, went, back, to, the, garden, .]   \n",
       "1986   [John, went, back, to, the, office, .]   \n",
       "1982   [John, journeyed, to, the, bedroom, .]   \n",
       "1979   [John, travelled, to, the, hallway, .]   \n",
       "\n",
       "                                                    tag  \\\n",
       "1995  [(John, NNP), (went, VBD), (back, RB), (to, TO...   \n",
       "1989  [(John, NNP), (went, VBD), (back, RB), (to, TO...   \n",
       "1986  [(John, NNP), (went, VBD), (back, RB), (to, TO...   \n",
       "1982  [(John, NNP), (journeyed, NN), (to, TO), (the,...   \n",
       "1979  [(John, NNP), (travelled, VBD), (to, TO), (the...   \n",
       "\n",
       "                       extracted  \n",
       "1995       (John, went, bedroom)  \n",
       "1989        (John, went, garden)  \n",
       "1986        (John, went, office)  \n",
       "1982  (John, journeyed, bedroom)  \n",
       "1979  (John, travelled, hallway)  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "person_statements_recent('John', n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Find the history of visitors for a room"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def find_room_visitors(room):\n",
    "    '''Find the list of visitors a room has, ordered by recency'''\n",
    "    \n",
    "    query = '''\n",
    "        MATCH (r:RELATION)-->(o:OBJECT {name:$name})\n",
    "        RETURN count(r) AS count\n",
    "    '''\n",
    "    \n",
    "    session = driver.session()\n",
    "    record = session.run(query, name=room).single()\n",
    "    \n",
    "    return record['count']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just for fun, we can find out how many times a room has been visited. \"How many times has the office been visited?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "334"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "find_room_visitors('office')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Calculate an Accuracy Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_answers(row):\n",
    "    '''Given an input row merge the statement in the graph, \n",
    "    or query the graph if it is a question'''\n",
    "    if row.type == 'S':\n",
    "        subject,relation,obj = row.extracted\n",
    "        session.run(v3_query, subject=subject, relation=relation, obj=obj)\n",
    "        return ''\n",
    "    elif row.type == 'Q':\n",
    "        person = row.extracted\n",
    "        # WARNING: do not consume the result (e.g., call .consume() or .single()) \n",
    "        # until the entire iteration is done.\n",
    "        # Failure to do so may cause the queries to be VERY slow!\n",
    "        return find_person(person)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start all over, and run through the entire dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "reset_db()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "session = driver.session()\n",
    "results = data_qa1.apply(get_answers, axis=1)\n",
    "results = [x for x in results if x != '']\n",
    "predicted = [result.single()['obj'].get('name') for result in results]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `predicted` array contains the predicted answer to each question.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bathroom', 'hallway', 'hallway', 'office', 'bathroom']"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `actual` array contains the actual answers to all questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "actual = list(data_qa1[data_qa1.type == 'Q'].answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bathroom', 'hallway', 'hallway', 'office', 'bathroom']"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "actual[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(actual, predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "And just like that, we get an accuracy of 100%. Of course, this dataset is very simple, but the graph we created can generalize to any statements of the form, `(subject, relation, object)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}