Schedule

Make sure to reload this page to ensure you’re seeing the latest version. Readings should be done before the indicated class. Readings include:

INLP Eisenstein, Intro to NLP, online text (or book version)
JM = Jurafsky and Martin, Speech and Language Processing, 3rd edition draft chapters

Learning for language

Mon 2/1: Introduction

Slides

Wed 2/3 (cancelled)

Mon 2/8: Linear classification

Slides (annotated)
Reading: INLP chapters 1 and 2

Wed 2/10: Nonlinear (neural net) classification

Slides (annotated)
Reading: INLP chapter 3

Mon 2/15: Linguistic classification and annotation

Slides (annotated 2/15) (annotated 2/17)
Demo: annotations & agreement rates spreadsheet
Reading: INLP chapter 4

Wed 2/17: Unsupervised learning

Annotated slides
Reading: INLP chapter 5

Mon 2/22: Word embeddings

Annotated slides
Reading: INLP chapter 14

Wed 2/24: No class

Neural Structure

Mon 3/1: Anna Rumshisky seminar on interpreting BERT

We are cancelling the main lecture for a very relevant seminar talk -- so please go to the seminar, if you can! 2-3pm, Anna Rumshisky from UMass Lowell, "What has your model learned? Interpreting the behavior of pre-trained Transformers." See Slack or seminars@cs for the Zoom URL.
We will have a 3:15pm session for any discussion of the talk, and to talk about the literature review.

Wed 3/3: Language Models

Annotated slides
Reading: INLP chapter 6

Mon 3/8: RNN LMs & transfer learning (ELMO)

Slides
Due: Reading Review #2. Read and review the paper for one of the tasks that the ELMO paper discusses. In addition to the usual review format, comment on why you think it may be possible that ELMO gives useful information for the task.
Due: the HW1 checkpoint.
Reading: Peters et al., 2018 (the ELMO paper)
Optional reading: Linzen et al., 2016 (LSTMs and long-distance syntax)

Tues 3/11: Notes from office hours

Wed 3/10: Attention and the Transformer Model

Annotated slides
Reading: Vaswani et al., 2017 ("Attention is All You Need")
Suggested: Michael Collins lecture (intro is great; Part 3 is on transformers), Illustrated Transformer (blogpost), and for less terrible notation, see the NamedTensor "Transformer" section (3.2) (or even the highly detailed Annotated Transformer (Rush 2018) paper)

Fri 3/12: Notes from office hours.

Mon 3/15: BERT

Annotated slides
Reading: Devlin et al. 2019, the BERT paper. Also useful: Devlin's slides (there are some other variants of it on the web).
Reading, esp. for Wed: Rogers et al. 2020, the "BERTology" paper, co-authored by Anna Rumshisky (who spoke at the March 1 seminar). There's lots here which we'll return to later on.

Wed 3/17: BERT & friends

Slides
Reading Review #3 (moved to being due Wed, not Mon): choose and review one of the 150 (!) BERT interpretation papers discussed by Rogers et al.

Syntax & Sequential Structure

Mon 3/22, Wed 3/24: Sequence Models

INLP chapters 7 and 8
3/22 annotated slides: sequence labeling, HMMs, forward algorithm.
3/24 annotated slides: Neural unsup. HMM, HMM vs. RNN, Viterbi, touch on CRFs.
3/22, 3/24 worksheet: forward/backward, Viterbi.

Fri 3/26: Lit review due

Mon 3/29, Wed 3/31: Constituent Syntax

3/29 annotated slides
3/31 annotated slides
INLP chapter 10
Linzen et al. 2016, LSTMs on subject-verb agreement
Tenney et al. 2019, "BERT Rediscovers the Classical NLP Pipeline." (See their cited ICLR 2019 paper for the constituency prediction setup.)

Fri 4/2: Project proposal due

Mon 4/5, Wed 4/7: Dependency Syntax

INLP chapter 11
Hewitt and Manning 2019, "A Structural Probe for Finding Syntax in Word Representations."

Semantics & Meaning

Mon 4/12: Colab demo and NLI

Demo of Google Colab and playing around with pre-trained BERT
Lecture on Natural Language Inference:
- Lecture notes
- Bowman et al. 2015, A large annotated corpus for learning natural language inference
- INLP, 12.1-12.2: Logical semantics, Meaning and denotation, Logical representations

Wed 4/14 is off: Well-Being Wednesday. Weather looks nice!

Mon 4/19: Coreference

Slides

Wed 4/21: No lecture

Social Factors in NLP

Mon 4/26: Gender

Slides
General review: Blodgett et al. 2020, Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Gender and...
- ...word embeddings: Bolukbasi et al. (2016), Caliskan et al. (2016)
- ...coreference: Rudinger et al. (2018), Zhao et al. (2018)
- ...translation: Stanovsky et al. (2019)
many other works; see e.g. Second Workshop on Gender Bias in NLP.

Wed 4/28: Dialects

Slides
African-American English: Blodgett et al. 2016, Demographic Dialectal Variation in Social Media: A Case Study of African-American English
Indian English: Demszky et al. 2020, Learning to Recognize Dialect Features
Arabic, on dialect identification: Zaidan and Callison-Burch 2006, Salameh et al. 2018
See also the Workshops on NLP for Similar Languages, Varieties, and Dialects; e.g. VarDial 2020

Fri 4/30: Progress Report due.

Mon 5/3: In-class presentations!

May 12: Final Report and HW3 due.