Natural language processing (NLP) is a critical component of modern information systems. The confluence of deep learning, massive datasets, and hardware accelerators has resulted in systems that approach and in some cases exceed human level performance on benchmark language tasks. In this course we will explore three principal questions: 1) how do we frame language understanding as a tractable statistical inference problem, 2) what are the current state-of-the-art modeling approaches, and 3) how are NLP systems designed and evaluated. In exploring these questions the course exposes students to the foundational math, implementation methods, practical applications, and current research directions in the field. The course will focus less on task-specific feature engineering in order to provide a more complete exposition of the methods that have proven to be most successful in real world applications such as web search, recommender systems, voice and text based chatbots, and machine translation. This course assumes a basic understanding of linear algebra, probability theory, first order optimization methods, and neural networks.
- Jacob Eisenstein. Natural Language Processing
- Dan Jurafsky, James H. Martin. Speech and Language Processing (3rd ed. draft)
Students are expected to adhere to all policies in the Georgetown Honor System Handbook, and be aware of Title IX/Sexual Misconduct. We take both of these matters seriously. All work submitted in this class is expected to be your own. In the case of code it is very common to use subroutines written by other people to perform specific tasks; this is OK but you must comment your code when doing so. Each student is required to acknowledge that they are aware of all course policies by completing the course policy quiz. In person attendance is highly recommended for students in section II.
The course grade is broken down in the table below. The course grade consists of 1000 total pts. The grading scale and components are listed in the tables below.
| Score | Letter Grade |
|---|---|
| 91.5 |
A |
| 89.5 |
A- |
| 87.99 |
B+ |
| 81.5 |
B |
| 79.5 |
B- |
| Component | Weight | Format | Section I Due Date | Section II Due Date |
|---|---|---|---|---|
| Quiz 1 | 3% | individual | Sep 08 | Sep 08 |
| Quiz 2 | 4% | individual | Sep 26 | Sep 22 |
| Quiz 3 | 4% | individual | Nov 15 | Nov 15 |
| Assignment 1 | 10% | groups <= 3 | Sep 26 | Sep 22 |
| Assignment 2 | 10% | individual | Oct 10 | Oct 06 |
| Assignment 3 | 10% | individual | Oct 30 | Oct 30 |
| Assignment 4 | 10% | individual | Nov 22 | Nov 22 |
| Labs | 20% | individual | Dec 08 | Dec 08 |
| Final Project | 25% | groups of <= 3 | Dec 08 | Dec 08 |
Note: All submissions are due 11:59pm EST on the dates listed. Submissions turned in more than 24 hours late will be assessed a one-time 15% penalty.
Lab Grading: Labs are due at the end of the semester (except for Lab-01, which only covers onboarding related material and is due Sep 08). This flexibility is not license to fall behind; the goal is for you to spend the time that is necessary to familiarize yourself with the models, algorithms, and design patterns covered in this course, not test how quickly you can learn the material on a cursory level. There will be 10 graded labs each worth 2%, and one extra credit lab. Each lab task will be graded on a complete/incomplete basis, however incomplete tasks will not necessarily be penalized provided that effort was made to complete them (which we can glean from your git commit history and the questions you ask).
Final Project Grading: The rubrik for the final project will be included in the project description released during Lab-07.
Students will submit assignments, labs, quizzes, and the project to private forks of this repository which the instructor and TAs will have access to. Setup instructions are here. If you are viewing this on canvas, you can view these instructions in Module-1: github-setup.html.
Students of this course will have access to recorded lectures and other large files via Georgetown's Box account, which is free for Faculty and Students. The course Box link is here.
| Section I | Section II | Lecture | Description | Course Materials |
|---|---|---|---|---|
| Aug 25 | Aug 26 | 1 | Introduction to NLP and deep learning - Course overview - Math refresher Lab-01 - Github onboarding - Python environment setup [Lecture] [Lab] |
Suggested Reading: [Eisenstein, Appendix A, B.1, B.2] [Linear algebra review] [Probability theory review] [Jurafsky & Martin, Chp 7] [Git+GitHub tutorial] |
| Aug 30 | Sep 02 | 2 | Text Classification: Part I - Bag-of-words (BOW) - Decision boundaries - The curse of dimensionality - The perceptron classifier - The linear SVM classifier Lab-02 - Text Normalization - Introduction to Spacy [Lecture] [Lab] |
Suggested Reading: [Eisenstein, Chps 2.1, 2.2, 2.5] [Jurafsky & Martin, Chp 2.4,4] [Rennie et al., ICML 2003] [Joachims, 1998] |
| Sep 08 | Sep 08 | Quiz 1 due 11:59pm EST Lab-01 Due 11:59pm EST |
[Quiz link ] [Lab link] |
|
| Sep 13 | Sep 09 | 3 | Text Classification: Part II - Maximum likelihood estimation - Naive Bayes' classifier - Softmax regression Lab-03 - Linear text classification on DBpedia dataset [Lecture] [Lab] |
Suggested Reading: [Eisenstein, Chps 2.3, 2.4] |
| Sep 20 | Sep 16 | 4 | Distributional Semantics: Part 1 - TF-IDF, PMI - Low-rank factorization - Topic modeling Lab-04 - Extracting topics from news feeds [Lecture] [Lab] |
Suggested Reading: [Ramos, 2003] [LSI overview] [SVD tutorial] [Lee et al., 2000] [Bei, et al., JMLR 2003] [Gensim website] |
| Sep 26 | Sep 22 | Quiz 2 due 11:59pm EST | [Quiz link] | |
| Sep 26 | Sep 22 | Assignment 1 due 11:59pm EST | [Assignment link] | |
| Sep 27 | Sep 23 | 5 | Distributional Semantics: Part 2 - Distributional hypothesis - Skip-gram & CBOW models - Word2Vec Lab-05 - Visualizing Tweets with Word2Vec [Lecture] [Lab] |
Suggested Reading: [Eisenstein, Chp 14] [Mikolov et al., 2013] [Blogpost on tSNE, UMAP] |
| Oct 04 | Sep 30 | 6 | Language Modeling: Part 1 - N-grams, sparse data & smoothing - Markov models Lab-06 - Build a semantic matching service with StarSpace and Virtex [Lecture] [Lab] |
Suggested Reading: [Eisenstein, Chp 6.1-6.2] [Jurafsky & Martin, Chps 3, 8] [Bojanowski et al., 2017] [Wu et al., 2017] [FastText] |
| Oct 17 | Oct 06 | Assignment 2 due 11:59pm EST | [Assignment link] | |
| Oct 18 | Oct 07 | 7 | Language Modeling: Part 2 - Introduction to neural language models - Next word prediction - Downstream tasks - Evaluation metrics Lab-07 (not graded) - Course project kick-off [Lecture] [Project] |
Suggested Reading: [Eisenstein, Chp 3] [NLP Progress website] [Huggingface Datasets] [Wang et al., 2018] |
| Oct 25 | Oct 14 | 8 | Neural Network Architectures for Sequences: Part I - Convolutional filtering - RNNs, Seq2Seq models - Siamese nets Lab-08 - Introduction to Huggingface - Train OpenNMT on en-de dataset [Lecture] [Lab] |
Suggested Reading: [Jurafsky & Martin, Chp 9] [Eisenstein, Chp 6.3] [Review of neural network architectures] [Overview of SGD algorithms] [Kim et al., 2016] [Minaee et al., 2020 (Sec. 2.8)] |
| Oct 30 | Oct 30 | Assignment 3 due 11:59pm EST | [Assignment link] | |
| Nov 01 | Oct 21 | 9 | Neural Network Architectures for Sequences: Part II - Masked language model training - Attention - Transformers (BERT, GPT) Lab-09 - Finetune BERT on DBpedia, CoNLL-2003, SQuADv2 [Lecture] [Lab] |
Suggested Reading: [Bahdanau et al., 2015] [Wu et al., 2016] [Vaswani et al., 2017] [Devlin et al., 2019] [Brown et al., 2020] [Solaiman et al., 2021] |
| Nov 08 | Oct 28 | 10 | Speech Processing Part I: A Primer on Speech and Audio Applications A Primer on Speech and Audio Applications Guest: Tarek Lahlou Distinguished Data Scientist, Walmart Labs Lab-10 (not graded) - Project work [Lecture] |
Suggested Reading: [Deepmind blog post] [Donahue et al., ICLR 2021] [Luo et al., 2021] |
| Nov 15 | Nov 04 | 11 | Speech Processing Part II: Speech Recognition - RNN Transducers, CTC - Transformer based models Lab-11 - Finetune E2E RNNT on your own voice [Lecture] [Lab] |
Suggested Reading: [Graves, 2012] [He et al., 2018] |
| Nov 10 | Nov 10 | Project proposals due 11:59pm EST | [Project description link] | |
| Nov 22 | Nov 11 | 12 | Dialogue Systems: Part I - Dialogue management - Entity extraction, slot filling - Policy based dialogue Lab-12 - Introduction to Rasa [Lecture] [Lab] |
Suggested Reading: [Jurafsky & Martin, Chp 24] [Bocklisch et al., 2017] |
| Nov 15 | Nov 15 | Quiz 3 Due 11:59pm EST | [Quiz Link] | |
| Nov 29 | Nov 18 | 13 | Dialogue Systems: Part II Introduction to Jarvis Guest: Oluwatobi Olabiyi Senior Research Manager, NVIDIA Lab-13 (not graded) - Project work [Lecture] |
|
| Nov 22 | Nov 22 | Assignment 4 due 11:59pm EST | [Assignment link] | |
| Dec 06 | Dec 02 | 14 | - REST, HTTP, RPCs - Containerization & cloud deployment - Production model serving - Deploy an embedding service on a VM [Lecture] [Lab] Final Project Presentations - Project presentations |
[Schedule / signup sheet] |
| Dec 08 | Dec 08 | Labs 1-6,8-10,12,14 due 11:59pm EST Final projects due 11:59pm EST |
||
| TBD | TBD | Final project poster session & demos |