examples/full-auto.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "In this notebook, we will use ASTrED in full-auto mode. That means that tokenisation, parsing, and word alignment all\n",
    " happen automatically. This is easy, but slow and likely less accurate than manual annotation. I would especially\n",
    " encourage you to use manual word alignments. But in this example, we show that you _can_ do it all automatically,\n",
    " which may be useful for large parallel corpus studies."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!pip install astred[stanza]\n",
    "!pip install git+https://github.com/BramVanroy/awesome-align.git@astred_compat"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from astred import AlignedSentences, Sentence"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "By default, the library assumes that the text that you provide is pretokenised and that words are separated by spaces.\n",
    " If that is not the case, we need to set `is_tokenised` to `False`.\n",
    "\n",
    "In the cell below, you'll notice that we do not provide any word alignment information to the `AlignedSentences`\n",
    " constructor. What is actually happening, is that - if no word alignments are provided -, an automatic aligner is\n",
    " instantiated which aligns the given source and target tokens. To do that we rely on a fork of AwesomeAlign\n",
    " (see the README), which is a multilingual, neural aligner.\n",
    "\n",
    "Running the command below for the first time may take a long time because stanza is downloading the neural models\n",
    " behind the scenes and Awesome Align is downloading the applicable transformer alignment model."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "sent_en = Sentence.from_text(\"Yesterday, I ate some cookies.\", \"en\", is_tokenized=False)\n",
    "sent_nl = Sentence.from_text(\"Ik at gisteren wat koekjes.\", \"nl\", is_tokenized=False)\n",
    "\n",
    "aligned = AlignedSentences(sent_en, sent_nl)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As you can see below, these alignments are good - but not great. All alignments are correct, but the alignment between\n",
    "\"Yesterday\" and \"gisteren\" is missing. The tokenizer did a perfect job, however!"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "print(aligned.src.text)\n",
    "print(aligned.tgt.text)\n",
    "print(aligned.giza_word_aligns)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also display the dependency trees to have a look at how well the automatic parser did. `stanza`\n",
    " (the parser) did a perfect job. The difference between `obl` for \"Yesterday\" and `advmod` for \"gisteren\" is open\n",
    " for discussion: `obl` is used for noun (phrases) and `advmod` for adverbs. Even on a theoretical level you can debate\n",
    " whether \"yesterday\" and \"gisteren\" are nouns or adverbs, but I'll leave that up to the theorists.\n",
    "\n",
    "Note how the trees display both the text and the dependency relation? You can specify whichever attribute of a `Word`\n",
    " that you want to (e.g. `upos`, `id`, `head` and so on) to `attrs` and it will be included in the tree."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# This cell does not work on remote environments such as Colab\n",
    "# Un-comment to try it on your local machine\n",
    "# from nltk.tree import Tree as NltkTree\n",
    "# from IPython.display import display\n",
    "#\n",
    "# display(NltkTree.fromstring(sent_en.tree.to_string(attrs=[\"text\", \"deprel\"])))\n",
    "# display(NltkTree.fromstring(sent_nl.tree.to_string(attrs=[\"text\", \"deprel\"])))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Separate aligner\n",
    "\n",
    "As a default, the aligner will make use of the pretrained model `bert-base-multilingual-cased`. This will be downloaded\n",
    " automatically behind the scenes. However, you may choose to train/finetune your own model, or download\n",
    " [pre-existing ones](https://github.com/neulab/awesome-align#model-performance), and use that instead. If that is the\n",
    " case, you can instantiate an aligner from-scratch and pass that to the `AlignedSentences` constructor, too! This way,\n",
    " the provided aligner will be used instead of the default one that uses `bert-base-multilingual-cased`."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from astred import Aligner\n",
    "\n",
    "# kwargs can contain some options specific to Awesome Aligner. Most important is probably the use of a GPU. By default\n",
    "# a GPU will be used if it is available.\n",
    "kwargs = {\"no_cuda\": True}\n",
    "# Provide directory that contains the pytorch_model.bin and other files\n",
    "# In this example, this code will not work of course because no custom model is given\n",
    "aligner = Aligner(r\"C:\\path\\to\\your\\model\\dir\", **kwargs)\n",
    "\n",
    "sent_en = Sentence.from_text(\"Yesterday, I ate some cookies.\", \"en\", is_tokenized=False)\n",
    "sent_nl = Sentence.from_text(\"Ik at gisteren wat koekjes.\", \"nl\", is_tokenized=False)\n",
    "\n",
    "aligned = AlignedSentences(sent_en, sent_nl, aligner=aligner)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If no `aligner` is provided, a class variable `AlignedSentences._aligner` will contain a default aligner\n",
    " that is used by all `AlignedSentences` instances. If you do not wish to use this default aligner, you can use the\n",
    " method above."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}