Schedule

INLP = Eisenstein (2019) textbook [pdf] [book site]  

W 1/22: Intro

M 1/27: Linear text classification

• INLP ch. 1 and 2

W 1/29: Nonlinear classif

• INLP ch. 3

M 2/3: Ling classif., annotations, stat. tests

• INLP ch. 4

W 2/5: Unsupervised learning

• INLP ch. 5

M 2/10: Language models

• INLP ch. 6

W 2/12: Recurrent neural networks

• INLP ch. 6, cont'd

T 2/18: Word embeddings and ELMo

• INLP ch. 14
• Peters et al. 2018

W 2/19: Attention/Transformers

• Vaswani et al. 2017
• Also: Michael Collins lecture (intro is great; Part 3 is on transformers)
• Illustrated Transformer (blogpost) and Annotated Transformer (Rush 2018)

M 2/24: BERT, part 1

• Devlin et al. 2018

W 2/26: BERT, part 2: reaction/refinements/madness

• New: Rogers et al. 2020, A Primer in BERTology (reviewing work trying to understand what BERT thinks)
• BERT variants
• Liu et al. 2019, RoBERTa - a "cleanup" paper.  Very informative to read after the original BERT paper.
• Joshi et al. 2020, SpanBERT - a variant
• Lan et al. 2020, ALBERT - yet another.
• Lee et al. 2019, BioBERT (and github)
• Fine-tuning
• Dodge et al. 2020 (in review? tweets) - Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping.
• Peters et al. 2019 - freeze vs fine-tuning on ELMo and BERT
• Other questions
• Michel et al. 2019 - Are 16 heads better than 1?
• Pires et al. 2019 - How Multilingual is Multilingual BERT?
• Probing BERT
• Tenney et al. 2019, BERT Rediscovers the Classical NLP Pipeline
• Hewitt and Manning 2019, dependency parse probe
• McCoy et al. 2019 (under review?), BERT's weaknesses for NLI

M 3/2: Sequence labeling (I)

• INLP ch. 7

W 3/4: Sequence labeling (II)

• INLP ch. 8

M 3/9: Syntax parsing (I)

• INLP ch. 10

W 3/11: Syntax parsing (II)

• INLP ch. 11
• Hewitt and Manning (2019) - syntactic dependencies in BERT
• This was on Zoom

[3/16-3/20: Spring Break]

Switch to Zoom

M 3/23: General discussion and COVID-19 papers dataset

[W 3/25: Cancelled]

M 3/30: Logical semantics -- INLP ch. 12

• Cool video: demo of SHRDLU https://www.youtube.com/watch?v=bo4RvYJYOzI

W 4/1: Predicate-argument semantics -- INLP ch. 13

M 4/6: Reference resolution -- INLP ch. 15

W 4/8: Is my result statistically significant? Statistics and evaluation -- INLP 4.4

M 4/13: Bootstrapping statistics - demo

W 4/15: Misc.

[No class on 4/20]

W 4/22: Ethics and NLP

M 4/27: Presentations

W 4/29: Presentations