Assignments

HW1, released 9/5, due 9/14 on Gradescope. [colab] [pdf]
HW2 - released 9/19, due Wed 10/1 on Gradescope.
HW3 - TBA
Seminar extra credit: Attend a talk from the 692L seminar series, or another NLP-related research talk at UMass approved by instructor. Write up the talk in several paragraphs, described in this google form; submit through that form. Talk write-ups count toward the exercise component of your grade.

Schedule

Make sure to reload this page to ensure you're seeing the latest version. Readings should be done before lecture. Note Jurafsky and Martin (JM) readings refer to the August 2025 edition of their online SLP3 draft.

Week 1: Introduction and Language Modeling

9/2: Course Introduction [slides]
9/4: N-Gram LMs [slides]
Reading: JM ch. 2, Words and Tokens, sections 2.1-2.2
Reading: JM ch. 3, N-gram LMs (all sections)
Reading: Bengio et al., 2003, "A Neural Probabilistic Language Model"
Optional: General blog post on language modeling (Lena Voita) (many of the topics, like generation, we'll return to later)

Week 2: Embeddings and Text Classification

9/9: Neural LMs and Word Embeddings [slides]
9/11: More Word Embeddings [slides], Text Classification [slides], sentiment keywords demo [colab]
Reading: see again Bengio et al. 2003
Reading: JM ch. 5, Embeddings
Reading: Landauer and Dumais, 1997, "Solution to Plato's Problem..." Psych. Review. Cognitive science perspective on learning word embeddings from text data. Experiments with an SVD-based embedding model, instead of the current neural-style logistic log-loss gradient learning approach.
Reading: JM ch. 4, Logistic Regression and Classification
Optional: Pang et al., EMNLP 2001 (supervised learning for sentiment classification)

Week 3: Neural Networks: MLPs, RNNs, Learning

9/16: Intro to Neural Networks [slides]
9/18: Backpropagation and NN Learning [slides]
- SGNS gradient math note, after lecture
Reading: JM ch. 6, Neural Networks
Reading: JM ch. 13, Recurrent NNs
Optional: Pascanu et al., ICML 2013, on vanishing gradients in RNNs
Optional: Andrej Karpathy blogpost (~2014?) with a coding-based explanation of backprop

Week 4: Large language models

9/24: Lecture on zoom (link from home page). Part 1: Course projects. Part 2: LLMs (ELMo and data)
9/26: Lecture cancelled (instructor travel)
Reading: JM ch. 7, Large Language Models

Week 5: Attention and Transformers

9/30: Transformers [slides]
10/2: More Transformers [slides]
Reading: JM ch. 8, Transformers
Optional readings on Transformers and Transformer-based generative LMs:
- Cosma Shalizi notes in Transformers, including the kernel smoothing viewpoint (Tsai et al., EMNLP 2019) and many varied references
- Blogpost: Illustrated GPT-2
- OpenAI's GPT line of work consists of a series of non-peer-reviewed tech reports, with progressively less technical detail; for example, GPT (Radford et al. 2013), GPT-2 (Radford et al. 2019), GPT-3 (Brown et al. 2020), and InstructGPT (Ouyang et al. 2022), the last of which was used in the original December 2022 version of ChatGPT.
- Blogpost: "Let's reproduce GPT-2", Karpathy, July 2024; short article and full reproduction of GPT-2 (to the extent it's possible), with implementation.
Optional readings on attention mechanisms:
- Attention mechanisms in NLP initially focused on machine translation, where naturally correspond to earlier "alignment"-based methods.
  - Bahdanau et al., ICLR 2015, Neural Machine Translation by Jointly Learning to Align and Translate
  - Luong et al., EMNLP 2015, Effective Approaches to Attention-based Neural Machine Translation
- Blog post on attention (Alammar)
- Video to understand self-attention (3Blue1Brown)

Week 6

Exercise #2 (due before 10/7 lecture): Load and run GPT-2 as a generator. (For example, follow HF docs' code snippets on Google Colab with a GPU enabled. Use any GPT-2 variant you prefer.) Experiment with different prompts (the left context before generation starts). Develop at least two prompts for (1) a factual question where the model makes a mistake, or does something wrong; (2) an open-ended question, or a prompt to start a story, news or social post, or other narrative text. For each prompt, generate at least twice from it, and comment on the quality of the result. Submit everything (for example, Colab PDF output, or a separate document as you prefer) to Gradescope, before the Tuesday 10/7 lecture. The purpose of this exercise is to try out a (non-instruction-tuned) Transformer LM; Come to lecture ready to discuss your experience with it.
10/7: Generative LLMs [slides]
Reading: JM ch. 7, Large Language Models (also assigned earlier, but we'll cover more of its material in lecture, in particular decoding)
Reading: Holtzmann et al., ICLR 2020, "The Curious Case of Neural Text Degeneration" (on nucleus sampling).
10/9: Midterm #1, in class. Practice midterm questions available on the Piazza Resources page.
Due next week: Project proposal, following these instructions / template.

Week 7

10/14: Fine-tuning LLMs [slides]
10/16: Annotations and tasks [slides]
Exercise #3 (annotations) [pdf]. Due on Gradescope before next lecture.
Reading: Eisenstein (INLP book), 4.5

Week 8: Instructions and alignment

10/21: [slides]
10/23: [slides]
Reading: JM ch. 9, Post-training
Optional: Lambert (2025), Reinforcement Learning from Human Feedback, online book/tutorial. The author also has done posts/videos/slidedecks on the topic; for example, their tutorial at ICML, summer 2023.
Optional: Ivison et al., NeurIPS 2024, Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback.

Week 9: Factuality and NLG Evaluation

10/28: NLI and Factuality [slides]
Exercise #4, Natural Language Inference (10/28 in-class activity). Submit PDF/scan on Gradescope by the 10/30 lecture.
10/30: MT, Generation and Evaluation [slides]
Exercise #5, MT (10/30 in-class activity). Submit PDF/scan on Gradescope before the next lecture. It's sufficient if you only find 5 new word translation pairs.

Weeks 10 and 11: Reasoning

11/4: no class (Election Day)
11/6: lecture
11/11: no class (Veteran's Day)
11/13: lecture

Week 12

11/18: lecture
11/20: Midterm #2, in class

Week 13

11/25: lecture
11/27: no class (Thanksgiving)

Weeks 14 and 15

12/2
12/4
12/9

After end of classes

12/17 (end of finals week): Final projects due. Template/directions to be posted.