Updating spaCy's Named Entity Recognition System.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Updating spaCy's Named Entity Recognition System"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pretrained models are simple to use, but they're unlikely to obtain state-of-the-art performance if your data differs even slightly from the type of data it was trained on. If state-of-the-art performance is what you're looking for, at some point you're going to want to train your own model. Luckily, spaCy allows this, too. In fact, spaCy offers us two options: it allows us to train a model from scratch, or to continue training its pretrained model with our own data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A toy example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SpaCy's pretrained named entity recognition model is pretty good, but of course, now and then it makes mistakes. Take a look at the sentence `Theresa May is a British politician serving as Prime Minister of the United Kingdom and Leader of the Conservative Party since 2016`. SpaCy successfully recognizes `British` as a nationality (NORP), `the United Kingdom` as a geo-political entity (GPE), `the Conservative Party` as an organization (ORG), and `2016` as a date. However, it does not recognize Theresa May as a person. Instead, it labels `Theresa` as an organization, and `May` as a date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tbody>\n",
       "<tr><td>Theresa     </td><td>B</td><td>ORG </td></tr>\n",
       "<tr><td>May         </td><td>B</td><td>DATE</td></tr>\n",
       "<tr><td>is          </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>a           </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>British     </td><td>B</td><td>NORP</td></tr>\n",
       "<tr><td>politician  </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>serving     </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>as          </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>Prime       </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>Minister    </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>of          </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>the         </td><td>B</td><td>GPE </td></tr>\n",
       "<tr><td>United      </td><td>I</td><td>GPE </td></tr>\n",
       "<tr><td>Kingdom     </td><td>I</td><td>GPE </td></tr>\n",
       "<tr><td>and         </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>Leader      </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>of          </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>the         </td><td>B</td><td>ORG </td></tr>\n",
       "<tr><td>Conservative</td><td>I</td><td>ORG </td></tr>\n",
       "<tr><td>Party       </td><td>I</td><td>ORG </td></tr>\n",
       "<tr><td>since       </td><td>O</td><td>    </td></tr>\n",
       "<tr><td>2016        </td><td>B</td><td>DATE</td></tr>\n",
       "<tr><td>.           </td><td>O</td><td>    </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import HTML, display\n",
    "import tabulate\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en\")\n",
    "text = \"Theresa May is a British politician serving as Prime Minister of the United Kingdom and Leader of the Conservative Party since 2016. \"\n",
    "\n",
    "doc = nlp(text)\n",
    "entities = [(t.text, t.ent_iob_, t.ent_type_) for t in doc]\n",
    "display(HTML(tabulate.tabulate(entities, tablefmt='html')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's fix this by giving the model some more training data. Obviously, we're not going to give it the exact sentence above &mdash; that would make the task just a bit too easy. Instead, we're going to use similar sentences with our target entity. We split up each of these sentences in its tokens, and provide each token with its correct label. In contrast to spaCy's output labelling scheme, these training labels follow the BILUO scheme. This means we don't just mark tokens and the Beginning and Inside of entities, but also tokens that make up an entity all by themselves (U), and those that are Last in the entity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_texts = [\n",
    "    ([\"Theresa\", \"May\", \"is\", \"determined\", \"to\", \"leave\", \"the\", \"EU\", \"in\", \"March\", \".\"],\n",
    "     [\"B-PERSON\", \"L-PERSON\", \"O\", \"O\", \"O\", \"O\", \"O\", \"U-ORG\", \"O\", \"U-DATE\", \"O\"]\n",
    "    ),\n",
    "    ([\"Theresa\", \"May\", \"says\", \"she\", \"will\", \"seek\", \"a\", \"pragmatic\", \"Brexit\", \"deal\", \".\"],\n",
    "     [\"B-PERSON\", \"L-PERSON\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]\n",
    "    ),\n",
    "    ([\"Theresa\", \"May\", \"vows\", \"to\", \"battle\", \"in\", \"Brussels\", \".\"],\n",
    "     [\"B-PERSON\", \"L-PERSON\", \"O\", \"O\", \"O\", \"O\", \"U-GPE\", \"O\"]\n",
    "    )\n",
    "]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of these training sentences, we make a spaCy document, reusing the vocabulary of the spaCy model we're using. Because we've already taken care of the tokenization, we also pass the tokens explicitly. Next, we combine this document with the correct labels in a so-called GoldParse object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.tokens import Doc\n",
    "from spacy.gold import GoldParse\n",
    "\n",
    "training_data = []\n",
    "for tokens, annotation in training_texts:\n",
    "    doc = Doc(nlp.vocab, words=tokens)\n",
    "    gold = GoldParse(doc, entities=annotation)\n",
    "    training_data.append((doc, gold))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're going to do the actual training. This means we're going to let our model see the labelled training data several times. For each of these so-called epochs, we shuffle the training data to avoid any form of bias, and update the model with the each of the training documents and its gold parse. We do this 10 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2fd164f6a7a04aac8fefe8143d74da9e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "\n",
    "for _ in tqdm(range(10)):\n",
    "    random.shuffle(training_data)\n",
    "    for doc, gold in training_data:\n",
    "        nlp.update([doc], [gold], drop=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now test the model on the same sentence as before. The output shows it still recognizes all the correct entities it found before, but now it has also identified `Theresa May` as a person. Hurray!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tbody>\n",
       "<tr><td>Theresa     </td><td>B</td><td>PERSON</td></tr>\n",
       "<tr><td>May         </td><td>I</td><td>PERSON</td></tr>\n",
       "<tr><td>is          </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>a           </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>British     </td><td>B</td><td>NORP  </td></tr>\n",
       "<tr><td>politician  </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>serving     </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>as          </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>Prime       </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>Minister    </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>of          </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>the         </td><td>B</td><td>GPE   </td></tr>\n",
       "<tr><td>United      </td><td>I</td><td>GPE   </td></tr>\n",
       "<tr><td>Kingdom     </td><td>I</td><td>GPE   </td></tr>\n",
       "<tr><td>and         </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>Leader      </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>of          </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>the         </td><td>B</td><td>ORG   </td></tr>\n",
       "<tr><td>Conservative</td><td>I</td><td>ORG   </td></tr>\n",
       "<tr><td>Party       </td><td>I</td><td>ORG   </td></tr>\n",
       "<tr><td>since       </td><td>O</td><td>      </td></tr>\n",
       "<tr><td>2016        </td><td>B</td><td>DATE  </td></tr>\n",
       "<tr><td>.           </td><td>O</td><td>      </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "text = \"Theresa May is a British politician serving as Prime Minister of the United Kingdom and Leader of the Conservative Party since 2016. \"\n",
    "\n",
    "doc = nlp(text)\n",
    "entities = [(t.text, t.ent_iob_, t.ent_type_) for t in doc]\n",
    "display(HTML(tabulate.tabulate(entities, tablefmt='html')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training an NER model on Dutch CONLL data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, however, you'll likely have more training data than just three examples with the same entity. Things become really interesting when you have access to a labelled data set of hundreds or more examples of several entity types: CVs that have been labelled with job titles and skills, medical documents that have been labelled with symptoms and diseases, etc.\n",
    "\n",
    "As an example, let's train a Named Entity Recognition model on the Dutch data that was collected for the [CoNLL-2002 Shared Task](https://www.clips.uantwerpen.be/conll2002/ner/). This data can be downloaded from Github."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-02-04 20:11:19--  https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/ned.train\n",
      "Resolving raw.githubusercontent.com... 151.101.36.133\n",
      "Connecting to raw.githubusercontent.com|151.101.36.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2377174 (2.3M) [text/plain]\n",
      "Saving to: 'data/ner/ned.train.6'\n",
      "\n",
      "data/ner/ned.train. 100%[=====================>]   2.27M  3.88MB/s   in 0.6s   \n",
      "\n",
      "2019-02-04 20:11:21 (3.88 MB/s) - 'data/ner/ned.train.6' saved [2377174/2377174]\n",
      "\n",
      "--2019-02-04 20:11:21--  https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/ned.testa\n",
      "Resolving raw.githubusercontent.com... 151.101.36.133\n",
      "Connecting to raw.githubusercontent.com|151.101.36.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 450785 (440K) [text/plain]\n",
      "Saving to: 'data/ner/ned.testa.6'\n",
      "\n",
      "data/ner/ned.testa. 100%[=====================>] 440.22K  --.-KB/s   in 0.1s   \n",
      "\n",
      "2019-02-04 20:11:21 (3.41 MB/s) - 'data/ner/ned.testa.6' saved [450785/450785]\n",
      "\n",
      "--2019-02-04 20:11:21--  https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/ned.testb\n",
      "Resolving raw.githubusercontent.com... 151.101.36.133\n",
      "Connecting to raw.githubusercontent.com|151.101.36.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 814282 (795K) [text/plain]\n",
      "Saving to: 'data/ner/ned.testb.6'\n",
      "\n",
      "data/ner/ned.testb. 100%[=====================>] 795.20K  2.97MB/s   in 0.3s   \n",
      "\n",
      "2019-02-04 20:11:22 (2.97 MB/s) - 'data/ner/ned.testb.6' saved [814282/814282]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/ned.train -P data/ner/\n",
    "!wget https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/ned.testa -P data/ner/\n",
    "!wget https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/ned.testb -P data/ner/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Dutch CoNLL data is formatted comes in the conll format (surprise, surprise). This means every line in the text files contains a token, and sentences are separated by empty lines. Every token consists of several tab-separated fields. For our purposes, we're just interested in the token itself (the first field), and its named entity label (the last field)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from operator import itemgetter\n",
    "\n",
    "train_file = \"data/ner/ned.train\"\n",
    "dev_file = \"data/ner/ned.testa\"\n",
    "test_file = \"data/ner/ned.testb\"\n",
    "\n",
    "def read_conll_file(f):\n",
    "    data = []\n",
    "    with open(f) as i:\n",
    "        sentences = i.read().strip().split(\"\\n\\n\")\n",
    "        \n",
    "    for sentence in sentences:\n",
    "        data.append([token.split() for token in sentence.split(\"\\n\")])\n",
    "\n",
    "    return data\n",
    "        \n",
    "train_data = read_conll_file(train_file)\n",
    "dev_data = read_conll_file(dev_file)\n",
    "test_data = read_conll_file(test_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Dutch CoNLL data contains the same entity types as spaCy's named entity pipe, but it wasn't part of the training data. As a result, spaCy's pretrained model performs so-so on the test data: it achieves an F-score of 63% for locations, 68% for organizations, 79% for persons and 54% for miscellaneous entities. Don't be fooled by the high average F-score: it's mainly due to the high accuracy of O tokens, which far outnumber the entities in our data. The total performance is not bad, but it's not very good, either."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report, precision_recall_fscore_support\n",
    "\n",
    "def evaluate(model, data, verbose=0): \n",
    "\n",
    "    ner = model.get_pipe(\"ner\")\n",
    "    \n",
    "    correct, predicted = [], []\n",
    "    for sentence in data:\n",
    "        tokens = [t[0] for t in sentence]\n",
    "        ent_labels = [t[2].split(\"-\")[-1] for t in sentence]\n",
    "        \n",
    "        doc = Doc(model.vocab, words=tokens)\n",
    "        ner(doc)\n",
    "        \n",
    "        pred_labels = [t.ent_type_ or \"O\" for t in doc]\n",
    "        correct += ent_labels\n",
    "        predicted += pred_labels\n",
    "        \n",
    "    if verbose:\n",
    "        print(classification_report(correct, predicted))\n",
    "    \n",
    "    return precision_recall_fscore_support(correct, predicted, average=\"micro\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5195\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         LOC       0.50      0.85      0.63       823\n",
      "        MISC       0.66      0.45      0.54      1597\n",
      "           O       0.99      0.99      0.99     63236\n",
      "         ORG       0.71      0.64      0.68      1433\n",
      "         PER       0.77      0.81      0.79      1905\n",
      "\n",
      "   micro avg       0.97      0.97      0.97     68994\n",
      "   macro avg       0.73      0.75      0.72     68994\n",
      "weighted avg       0.97      0.97      0.97     68994\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.9663159115285387, 0.9663159115285387, 0.9663159115285387, None)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp = spacy.load(\"nl\")\n",
    "evaluate(nlp, test_data, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now see what happens if we train a spaCy model specifically on the CoNLL data. To this goal, we'll convert the CoNLL data to spaCy documents and GoldParses like we did above. This means we have to convert its BIO labels to BILUO labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.gold import iob_to_biluo\n",
    "\n",
    "training_data = []\n",
    "for sentence in train_data:\n",
    "    tokens = [t[0] for t in sentence]\n",
    "    ent_labels = iob_to_biluo([t[2] for t in sentence])\n",
    "    doc = Doc(nlp.vocab, words=tokens)\n",
    "    gold = GoldParse(doc, entities=ent_labels)\n",
    "    training_data.append((doc, gold))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll now compare two different situations. First we'll train a new spaCy model from scratch. We do this by initializing a blank Dutch spaCy model with `spacy.blank(\"nl\")`. We'll add a named entity recognition pipe to it, and add the four entity labels in our training data. Then, we'll initialize the document for training. \n",
    "\n",
    "Second, we don't train a model from scratch, but we take the pretrained spaCy entity model and continue training it on our new training data. This means we can make use of everything the pretrained model has already learnt from its original training set. Because this model has seen much more data, we hope it will eventually give better results.\n",
    "\n",
    "Apart from the initialization stage, the training of these two models looks exactly the same. We disable all other pipes, and train the models for a maximum of 100 epochs. Whenever we achieve a new highest F-score on the development data, we save them. To avoid overfitting, we break the training cycle whenever we haven't been able to improve on the development F-score for three steps in a row. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.util import minibatch\n",
    "from pathlib import Path\n",
    "\n",
    "def train(train_docs, dev_data, output_dir, model=None, max_epochs=100): \n",
    "    \n",
    "    if not model: \n",
    "        model = spacy.blank(\"nl\")\n",
    "        ner = model.create_pipe(\"ner\")\n",
    "        model.add_pipe(ner, last=True)\n",
    "        for label in [\"PER\", \"LOC\", \"ORG\", \"MISC\"]: \n",
    "            ner.add_label(label)\n",
    "        model.begin_training()\n",
    "        \n",
    "    other_pipes = [pipe for pipe in model.pipe_names if pipe != 'ner']\n",
    "    fscore_history = []\n",
    "    patience=3\n",
    "        \n",
    "    with model.disable_pipes(*other_pipes):\n",
    "    \n",
    "        for i in range(max_epochs):\n",
    "            print(\"Epoch\", i)\n",
    "            losses = {}\n",
    "            random.shuffle(train_docs)\n",
    "            batches = minibatch(train_docs, size=32)\n",
    "            for batch in tqdm(batches):\n",
    "                docs, golds = zip(*batch)\n",
    "                model.update(\n",
    "                    docs,\n",
    "                    golds,\n",
    "                    drop=0.4,\n",
    "                    losses=losses)\n",
    "            print(\"Training Loss:\", losses)\n",
    "            \n",
    "            _, _, dev_f, _ = evaluate(model, dev_data)\n",
    "            print(\"Development F-score:\", dev_f)\n",
    "            \n",
    "            if len(fscore_history) > 0 and dev_f > max(fscore_history): \n",
    "                if output_dir is not None:\n",
    "                    output_dir = Path(output_dir)\n",
    "                    if not output_dir.exists():\n",
    "                        output_dir.mkdir()\n",
    "                    model.to_disk(output_dir)\n",
    "                    print(\"Saved model to\", output_dir)\n",
    "            \n",
    "            fscore_history.append(dev_f)\n",
    "            \n",
    "            if max(fscore_history) > max(fscore_history[-patience:]):\n",
    "                print(\"No improvement on development set. Stop training.\")\n",
    "                break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we train the completely new model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: Unnamed vectors -- this won't allow multiple vectors models to be loaded. (Shape: (0, 0))\n",
      "Epoch 0\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "94c94ba877cf4f3694fe0306d8785180",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 17.634281397026918}\n",
      "Development F-score: 0.955536135165912\n",
      "Epoch 1\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f9e258a80b4146b89f54e85332a24c01",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 8.124413278556299}\n",
      "Development F-score: 0.9652551574375678\n",
      "Saved model to models/spacy_ner_scratch\n",
      "Epoch 2\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0016501308724a1c8bb981459d6c4ce5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 5.07255810720055}\n",
      "Development F-score: 0.9674531924472339\n",
      "Saved model to models/spacy_ner_scratch\n",
      "Epoch 3\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1c77830b5924e55a78eeafeb3182c12",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 3.594017944427045}\n",
      "Development F-score: 0.9692539922141894\n",
      "Saved model to models/spacy_ner_scratch\n",
      "Epoch 4\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7858449e756942d481bd2722fe315a57",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 2.818861507985981}\n",
      "Development F-score: 0.9607266756706655\n",
      "Epoch 5\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a383146f484448d84e61a5ff909da0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 2.3161421975320895}\n",
      "Development F-score: 0.9678769100394587\n",
      "Epoch 6\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a049b60ff8f4ec182d33df8a22df82a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 1.934440403581468}\n",
      "Development F-score: 0.9677974629909165\n",
      "No improvement on development set. Stop training.\n"
     ]
    }
   ],
   "source": [
    "output_dir_scratch = \"models/spacy_ner_scratch\"\n",
    "train(training_data, dev_data, model=None, output_dir=output_dir_scratch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we continue training the existing model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6775cfb8304447c6b164855c94dce28e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 6.690205245732558}\n",
      "Development F-score: 0.9751860385053361\n",
      "Epoch 1\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dfde979ac40346928884f189395391f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 3.8389393237998024}\n",
      "Development F-score: 0.9784698498450782\n",
      "Saved model to models/spacy_ner_cntd\n",
      "Epoch 2\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32fb9ee1130548f9a4808c07d6181d3b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 2.788690348488808}\n",
      "Development F-score: 0.9790259791848733\n",
      "Saved model to models/spacy_ner_cntd\n",
      "Epoch 3\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c9de435e721c48a3b4ed46bb48eace2e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 2.0977905896412414}\n",
      "Development F-score: 0.9789200497868171\n",
      "Epoch 4\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "916a6a08965a4fe7ae824ec8c1fcdcf1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 1.7780554252080347}\n",
      "Development F-score: 0.9781520616509096\n",
      "Epoch 5\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bb2654ba30e8464eb9f99909a3f5af82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Loss: {'ner': 1.4171092849177291}\n",
      "Development F-score: 0.9782579910489658\n",
      "No improvement on development set. Stop training.\n"
     ]
    }
   ],
   "source": [
    "output_dir_cntd = \"models/spacy_ner_cntd\"\n",
    "train(training_data, dev_data, model=nlp, output_dir=output_dir_cntd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The F-scores we recorded on the development data already suggested that the continued model is indeed better than the completely new model: its development F-score lies around 1% higher. This is confirmed by the results on our testing data. \n",
    "\n",
    "First, our new model already scores better than spaCy's pretrained model on the CoNLL test data. This is particularly the case for the LOC and MISC entities, where its F-score lies 14% and 13% higher, respectively. This shows how important it is to train on in-domain data: although spaCy's pretrained model has seen more data than our CoNLL model, the higher similarity of the CoNLL training data to our testing data makes our new model perform much better. \n",
    "\n",
    "Second, the continued model goes one step further. It improves the F-score on the locations by another 8%, on the miscellaneous entities by 9%, on the organizations by 9%, and on the persons by 7%. As some parts of the training data are random (such as the random order in which we feed the data to the model), your mileage may vary, but the bigger patterns should be pretty similar. They demonstrate how the continued training is able to combine the strengths of the two approaches above: it still relies in part on the knowledge that was encoded in the pretrained model, but it has finetuned this model on our in-domain data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "********** Base Model **********\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         LOC       0.50      0.85      0.63       823\n",
      "        MISC       0.66      0.45      0.54      1597\n",
      "           O       0.99      0.99      0.99     63236\n",
      "         ORG       0.71      0.64      0.68      1433\n",
      "         PER       0.77      0.81      0.79      1905\n",
      "\n",
      "   micro avg       0.97      0.97      0.97     68994\n",
      "   macro avg       0.73      0.75      0.72     68994\n",
      "weighted avg       0.97      0.97      0.97     68994\n",
      "\n",
      "\n",
      "********** New Model **********\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         LOC       0.76      0.78      0.77       823\n",
      "        MISC       0.78      0.59      0.67      1597\n",
      "           O       1.00      1.00      1.00     63236\n",
      "         ORG       0.71      0.67      0.69      1433\n",
      "         PER       0.76      0.89      0.82      1905\n",
      "\n",
      "   micro avg       0.98      0.98      0.98     68994\n",
      "   macro avg       0.80      0.79      0.79     68994\n",
      "weighted avg       0.98      0.98      0.97     68994\n",
      "\n",
      "\n",
      "********** Transfer Model **********\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         LOC       0.88      0.82      0.85       823\n",
      "        MISC       0.83      0.70      0.76      1597\n",
      "           O       1.00      1.00      1.00     63236\n",
      "         ORG       0.78      0.78      0.78      1433\n",
      "         PER       0.86      0.93      0.89      1905\n",
      "\n",
      "   micro avg       0.98      0.98      0.98     68994\n",
      "   macro avg       0.87      0.85      0.86     68994\n",
      "weighted avg       0.98      0.98      0.98     68994\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.9825926892193524, 0.9825926892193524, 0.9825926892193524, None)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp_base = spacy.load(\"nl\")\n",
    "nlp_scratch = spacy.load(output_dir_scratch)\n",
    "nlp_cntd = spacy.load(output_dir_cntd)\n",
    "\n",
    "print(\"\\n********** Base Model **********\")\n",
    "evaluate(nlp_base, test_data, verbose=1)\n",
    "print(\"\\n********** New Model **********\")\n",
    "evaluate(nlp_scratch, test_data, verbose=1)\n",
    "print(\"\\n********** Continued Model **********\")\n",
    "evaluate(nlp_cntd, test_data, verbose=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, the continued training of a pretrained model is not without its challenges. Most importantly, you need to make sure that your model doesn't overfit on the new training data and loses its ability to label the type of data it was originally trained on. A related challenge is that of [catastrophic forgetting](https://explosion.ai/blog/pseudo-rehearsal-catastrophic-forgetting), which typically occurs when weights are shared between several NLP tasks. Still, when you get it right, finetuning an existing model is a powerful way of training high-quality model with a limited amount of data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}