Homework assignments

See Gradescope for the most up to date due date.

HW1, due Wed 9/25.
HW2, with multiple due dates; see PDF.
HW3 (colab notebook), due Mon 11/25.
HW4, due Wed 12/11.
Final reports are due Wed 12/18 (last day of finals).

Schedule

Lecture topics will appear below. Make sure to reload this page to ensure you’re seeing the latest version. Readings should be done before the indicated class. The main textbook is available free online:

JM = Jurafsky and Martin, Speech and Language Processing, 3rd edition, online version from August 2024. (Note chapters were pretty different before their Aug 2024 update.)
INLP = Eisenstein, Intro to NLP, online text (or book version; we'll have readings from this one occasionally, and it's recommended in general.)

Tue 9/3: Introduction

Thu 9/5: Words and regular expressions

Reading: JM ch. 2.1-2.4
Optional: Ken Church, "Unix for Poets"
Optional: RegexOne
Optional: David C. Zentgraf, "What Every Programmer Absolutely, Positively Needs To Know About Encodings And Charater Sets To Work With Text"
Tokenization demo
"Exercise" 1: Class survey (electronic only on Gradescope)

Tue 9/10 and Thu 9/12: Text Classification and Naive Bayes (annotated)

Reading: JM 4.1-4.6
Optional: INLP 4.1-4.3
In-class activity: Manual keywords.

Tue 9/17: Logistic Regression Classification (annotated)

Reading: JM ch. 5.1-5.5
Exercise 2: Logistic regression, on slide 12. (Gradescope makeup: make PDF for the answer for slide 12.)

Tue 9/24: Classification Evaluation (annotated)

Reading: JM 5.7
Reading: JM 4.7, 4.8, 4.10

Tue 9/24 + Thu 9/26: Annotations (annotated)

Reading: INLP, 4.5
Exercise 3: Interannotator agreement

Tue 10/1: Text Data and Final Project Overview

Thu 10/3: Syntax (I): Tagging and CFGs

Reading: JM 17.1-17.3 (Word Classes, POS Tagging, NER)
Reading: JM 18.1-18.5 (Context-Free Grammars); will revisit next week
Exercise 4: Parts of speech

Tue 10/8: Syntax (II): Grammar exercise session (slides)

Exercise 5: CFG
Extra reading: INLP Ch. 9, which has the exercise grammar.

Thu 10/10: Syntax (III): Dependency Parsing (annotated)

Reading: JM 19.1-19.2 (Dependency Relations, Transition-Based Dependency Parsing)
HW2 "Phase 2" is due on Friday.

Tue 10/15: No class (Monday class schedule)

Thu 10/17: N-Gram Language Models (annotated)

Reading: JM 3.1-3.7, N-grams, Evaluation, Sampling, Generalization, Interpolation, Entropy
Exercise 6: LM (Shannon game)

Tue 10/22 (annotated), Thu 10/24 (annotated): Word Embeddings

Reading: JM ch. 6, Vector Semantics and Embeddings
Reading: Garten et al., 2018, Dictionaries and distributions: Combining expert knowledge and large scale textual data content analysis: Distributed dictionary representation
Reading: Demszky et al., 2019, Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings

Tue 10/29: Midterm review

Practice midterm questions and topic listing
Partial solutions and blackboard photos on Piazza resources page

Thu 10/31: Midterm, in class

Tue 11/5: No class (Election Day)

Thu 11/7: Neural Networks in NLP

JM 7.1-7.4, Neural networks
JM ch. 8, RNNs and LSTMs
JM ch. 9, the Transformer model (more next week)

Tue 11/12, Thu 11/14: Transformers and BERT

Exercise 7: 11/14 in-class BERT demo/exercise with Google Colab. (in-class version)
JM ch. 11, masked LMs (i.e., BERT = MLM-pretrained Transformer encoder)
Rogers et al. 2020, "A Primer in BERTology: What We Know About How BERT Works", TACL
Manning et al. 2020, "Emergent linguistic structure in artificial neural networks trained by self-supervision", PNAS
T5 (Raffel et al., JMLR 2020)
Some alternate perspectives and references: Cosma Shalizi's notes on Transformers/LLMs

Tue 11/19, Thu 11/21: Generative LLMs

Transformer LLMs:

JM ch. 10, large language models
Blog post: The Illustrated GPT-2 (Alammar, 2018)
GPT-2 (Radford et al. 2019)
Nucleus sampling (Holtzman et al., ICLR 2020)

Prompting and such:

JM ch. 12: alignment, prompting, in-context learning
GPT-3 (Brown et al. 2020) and instruction tuning thereof (Ouyang et al. 2022); these represent, approximately, the original December 2022 ChatGPT.