examples/automatic-parsing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Providing tokenisation and word alignment\n",
    "\n",
    "This is probably the most common use-case for this library. You have data that is already tokenised and aligned,\n",
    " and you want to calculate the metrcics. That means that the text still needs to be parsed. In this example, under the\n",
    " hood the `stanza` parser processes the given, pre-tokenised input text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!pip install astred[stanza]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from astred import AlignedSentences, Sentence"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "First we need to create the source and target `Sentence` objects by passing space-sparated tokens and specifying which\n",
    " language the parser should use. In this case, English (en) and Dutch (nl). These parsed sentences can then be used to\n",
    " created an aligned object, but for that we need word alignments. These should be written in the Pharaoh format `i-j`\n",
    " where `i` is the index of a source token and `j` the index of the target token that it is aligned with. And that's it!\n",
    "\n",
    "(It does not make sense to have non-tokenized text alongside existing word alignments: every alignment unit must\n",
    " already correspond to one token, right?)\n",
    "\n",
    "Running the command below for the first time may take a long time because stanza is downloading the neural models\n",
    " behind the scenes."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "sent_en = Sentence.from_text(\"The show is billed as the museum 's largest ever .\", \"en\")\n",
    "sent_nl = Sentence.from_text(\"Dit is de duurste voorstelling ooit in het museum .\", \"nl\")\n",
    "aligns = \"0-0 0-2 1-4 2-1 3-1 4-1 5-7 6-8 7-6 8-3 9-5 10-9\"\n",
    "\n",
    "aligned = AlignedSentences(sent_en, sent_nl, word_aligns=aligns)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can now loop over the words in a sentence. Note that we explicitly ask for `Word`s that are not NULL words. If you\n",
    " want to iterate over all `Word`s, including NULL, then use `for word in aligned.src`. For each word you can also find\n",
    " the word(s) that it is aligned with, which allows for some interesting analyses."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "for word in sent_nl.no_null_words:\n",
    "\tfor aligned_word in word.aligned:\n",
    "\t\tprint(word.text, aligned_word.text, word.deprel, aligned_word.deprel)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "For each word you can also see whether some of its properties differ from its aligned word(s). For instance, in the\n",
    " example below we check for the Dutch word \"is\" whether its POS label has changed. `.changes()` returns a\n",
    " dictionary of an integer (word_id of aligned word) to a boolean (whether or not the label has changed)."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "verb_is = sent_nl[2]\n",
    "print(\"Dutch:\", verb_is.text, verb_is.upos)\n",
    "for aligned_id, change in sent_nl[2].changes(\"upos\").items():\n",
    "\tprint(\"Aligned:\", sent_en[aligned_id].text, sent_en[aligned_id].upos, change)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The library also provides word-based metrics such as sequence cross and SACr cross. For every sentence, we\n",
    " can iterate over its created groups. Terminology-wise, we use \"spans\" to refer to the groups. Those spans have\n",
    " their own properties. You can just get their text, but because spans are aligned to other spans (on the other side)\n",
    " you can get those as well (similar to above). It follows that spans can cross other spans when they move, so you can\n",
    " get their `cross` value as well.\n",
    "\n",
    "We have to iterate `no_null_*_spans` because NULL words form their own spans."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "for span in sent_en.no_null_seq_spans:\n",
    "\tfor aligned_span in span.aligned:\n",
    "\t\tprint(span.text, aligned_span.text, span.cross)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "For SACr groups you can additionally find the root node in that group. That is the node that is highest in the\n",
    " dependency tree. Below we first visualise the source tree and then show for each SACr span its root."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "sent_en = Sentence.from_text(\"Sometimes she asks me why I used to call her father Harold .\", \"en\")\n",
    "sent_nl = Sentence.from_text(\"Soms vraagt ze waarom ik haar vader Harold noemde .\", \"nl\")\n",
    "aligns = \"0-0 1-2 2-1 4-3 5-4 6-8 7-8 8-8 9-5 10-6 11-7 12-9\"\n",
    "\n",
    "aligned = AlignedSentences(sent_en, sent_nl, word_aligns=\"0-0 1-2 2-1 4-3 5-4 6-8 7-8 8-8 9-5 10-6 11-7 12-9\")\n",
    "\n",
    "for span in sent_en.no_null_sacr_spans:\n",
    "\tprint(span.text, span.root.text)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# This cell does not work on remote environments such as Colab\n",
    "# Un-comment to try it on your local machine\n",
    "# from nltk.tree import Tree as NltkTree\n",
    "# from IPython.display import display\n",
    "#\n",
    "# display(NltkTree.fromstring(sent_en.tree.to_string()))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}