Assignments

• Homework 0 released, due 2/4 on Gradescope
• Quiz 1 released, due 2/22
• Final project proposals due 2/25 on Gradescope, use this Overleaf template
• Quiz 2 released, due 3/7
• Homework 1 released, due 3/25 on Gradescope
• Quiz 3 released, due 3/11
• Midterm released on 3/30, due 4/1 at 8AM on Gradescope
• Quiz 4 released, due 4/19
• Quiz 5 released, due 5/6
• Homework 2 released, due 5/4
• Final project reports & code due 5/12, use this Overleaf template

Schedule

Make sure to reload this page to ensure you're seeing the latest version. Readings should be done before watching the corresponding lecture videos. See this page for materials (videos / slides / reading) from the Fall 2021 offering.

Week 1 (1/26): introduction

• • Course introduction // [video] // [slides]
  • No associated readings or weekly quiz!
  • HW 0 released here, due 2/4


• • Final projects:
  • • organize into groups by 2/7
    • proposal due 2/25 on Gradescope, use this Overleaf template

Week 2 (1/31, 2/2): language models

• • Language modeling // [video] // [slides]
  • [reading] Jurafsky & Martin, 3.1-3.5 (language modeling)
  • [reading] Jurafsky & Martin, 7 (neural language models)
  • [F2020 video]
  • [F2021 video]


• • Neural language models // [video] // [slides]
  • [reading] Neural language models (Bengio 2003)
  • [F2021 video]
  • [F2020 video] and colab notebook on implementing a simple neural LM

Week 3 (2/7-9): backpropagation, attention mechanisms

• • Backpropagation // [video] // [notes]
  • [optional reading] Andrej Karpathy's coding-based backpropagation post
  • [optional reading] Pascanu et al., ICML 2013 (vanishing gradients in RNNs)
  • [F2021 video]
  • See F2020 video timestamped here for derivation of gradient through linear layer


• • Attention mechanisms & self-attention // [video] // [slides] // [notes]
  • [reading] Bahdanau et al., ICLR 2015 (paper that introduced attention)
  • [optional reading] An easy-to-read blog post on attention
  • [F2021 video]
  • [F2020 video]

Week 4 (2/14-16): Transformers

• • Transformer language models // [video] // [slides] // [notes]
  • [reading] Vaswani et al., NeurIPS 2017 (paper that introduced Transformers)
  • [reading] An easy-to-read blog post on Transformer language models
  • Quiz 1 released, due 2/22


• • Transformers (cont'd) and transfer learning // [video] // [notes]
  • [reading] Deep contextualized word representations (Peters et al., 2018, "ELMo")
  • [reading] BERT: Pre-training of Deep Bidirectional Transformers... (Devlin et al., 2019)
  • [reading] Easy-to-read blog post on transfer learning in NLP

Week 5 (2/22-23): Transfer learning

• • BERT // [video] // [notes]
  • [optional reading] RoBERTa: more data (Liu et al., 2019)
  • [optional reading] ELECTRA: faster training (Clark et al., 2020)
  • [paper brought up in class] Should you mask 15% in Masked LM? (Wettig et al., 2022)
  • [paper brought up in class] Adapting pretrained LMs to offline RL (Reid et al., 2022)


• • BERT (cont'd)// [video] // [notes]
  • [reading] Exploring the Limits of Transfer Learning... (Raffel et al., JMLR 2020, "T5")
  • [optional reading] Nucleus sampling (Holtzmann et al., ICLR 2020)

Week 6 (2/28-3/2): Text-to-text transfer and decoding

• • T5 & LM decoding// [video] // [slides] // [notes]
  • [reading] Exploring the Limits of Transfer Learning... (Raffel et al., JMLR 2020, "T5")
  • [optional reading] Nucleus sampling (Holtzmann et al., ICLR 2020)


• • Tokenization // [video] // [slides]
  • [reading] Neural Machine Translation... with Subword Units (Sennrich et al., ACL 2016)
  • [reading] ByT5: Towards a token-free future... (Xue et al., 2021)
  • [optional reading] CANINE... Tokenization-free encoder... (Clark et al., 2021)
  • [optional reading] Charformer... (Tay et al., 2021)
  • Quiz 2 released, due 3/7
  • HW 1 released here, due 3/25

Week 7 (3/7-9): Prompt-based learning, evaluating text generation

• • Prompt-based learning // [video] // [slides] // [notes]
  • [reading] Pre-train, Prompt, and Predict... (Liu et al., 2021, survey paper)
  • [optional reading] Power of Scale for Prompt Tuning (Lester et al., EMNLP 2021)
  • [optional reading] Prefix tuning... Prompts for Generation (Li & Liang, ACL 2021)
  • Quiz 3 released, due 3/11


• • Evaluating text generation models // [video] // [slides]
  • [reading] Evaluation of text generation survey (Celikyilmaz et al., 2020)
  • [reading] BLEURT: robust metrics for text generation (Sellam et al., 2020)
  • [optional reading] Do massively pretrained LMs make better storytellers? (See et al., 2019)

Week 8 (3/21-23): Multilingual LMs, retrieval-augmented LMs

• • Multilingual transfer learning // [video] // [slides]
  • [reading] Beyond English-centric multilingual machine translation (Fan et al., 2020)
  • [reading] MAD-X: Multi-task cross lingual transfer (Pfeiffer et al., EMNLP 2020)


• • Retrieval-augmented LMs // [video] // [slides]
  • [reading] REALM: retrieval-augmented LMs (Guu et al., 2020)
  • [reading] Nearest neighbor machine translation (Khandelwal et al., ICLR 2021)
  • [optional reading] Hurdles to progress in long-form QA (Krishna et al., NAACL 2021)

Week 9 (3/28-30): Midterm exam

• • Midterm review // [video] // [notes]


• • No class on 3/30, midterm to be released at 8AM on 3/30, due 8AM on 4/1 via Gradescope

Week 10 (4/4-6): Efficient Transformers, commonsense reasoning

• • Efficient / long-range Transformers // [video] // [slides]
  • [reading] Survey of Efficient Transformers (Tay et al., 2020)
  • [reading] Routing Transformers (Roy et al., TACL 2020)
  • [optional reading] Do long-range LMs use long-range context? (Sun et al., EMNLP 2021)


• • Commonsense reasoning (pre-recorded guest lecture by Lorraine Li) // [video] // [slides]

Week 11 (4/11-13): Linguistic probe tasks, vision + language

• • Linguistic probe tasks (no in-class lecture!) // [video] // [slides]
  • [reading] What you can cram into a single $&!#* vector (Conneau et al., 2018)
  • [reading] Control probes (Hewitt & Liang et al., 2019)


• • Vision + language (no in-class lecture!) // [video] // [slides]
  • [reading] Visual-semantic alignments for image captioning (Karpathy & Fei-Fei, 2014)
  • [reading] Learning... visual models from NL supervision (Radford et al., 2021, "CLIP")
  • [reading] NLVR: A corpus of natural language for visual reasoning (Suhr et al., 2017)
  • Quiz 4 released, due 4/19
  • HW 2 released here, due 5/4

Week 12 (4/20): Model distillation

• • Model distillation // [video] // [slides]
  • [reading] Imitation attacks on MT systems (Wallace et al., 2020)
  • [reading] Thieves on Sesame Street! (Krishna et al., 2020)
  • [reading] Lottery ticket hypothesis for BERT (Chen et al., 2020)

Week 13 (4/25-27): Syntactic and semantic parsing

• • Syntactic parsing // [video] // [slides]
  • [reading] Jurafsky & Martin Ch. 13
  • [optional reading] Learning syntax from bracketings (Shi et al., NAACL 2021)
  • [optional reading] Unsupervised constituency parsing (Drozdov et al., NAACL 2019, "DIORA")


• • Semantic parsing // [video] // [slides]
  • [reading] WikiTableQuestions (Pasupat & Liang, 2015)
  • Quiz 5 released, due 5/6

Week 14 (5/2-4): Ethics in NLP & Scaling Laws of Large LMs

• • Ethics in NLP // [video] // [slides]
  • [optional readings] Yulia Tsvetkov's Ethics for NLP class


• • Scaling laws // [video] // [slides]
  • [reading] Scaling Laws for Neural Language Models (Kaplan et al., 2020)
  • [reading] Training Compute-Optimal Large Language Models (Hoffmann et al., 2022)
  • [reading] PaLM: Scaling Language Modeling with Pathways (Chowdhery et al., 2022)