Assignments

• HW1, released 9/5, due 9/14 on Gradescope. [colab] [pdf]
• HW2 - TBA
• HW3 - TBA

Schedule

Make sure to reload this page to ensure you're seeing the latest version. Readings should be done before lecture. Note Jurafsky and Martin (JM) readings refer to the August 2025 edition of their online SLP3 draft.

Week 1: Introduction and Language Modeling

• 9/2: Course Introduction [slides]
• 9/4: N-Gram LMs [slides]
• Reading: JM ch. 2, Words and Tokens, sections 2.1-2.2
• Reading: JM ch. 3, N-gram LMs (all sections)
• Reading: Bengio et al., 2003, "A Neural Probabilistic Language Model"
• Optional: General blog post on language modeling (Lena Voita) (many of the topics, like generation, we'll return to later)

Week 2: Embeddings and Text Classification

• 9/9: Neural LMs and Word Embeddings [slides]
• 9/11: More Word Embeddings [slides], Text Classification [slides], sentiment keywords demo [colab]
• Reading: see again Bengio et al. 2003
• Reading: JM ch. 5, Embeddings
• Reading: Landauer and Dumais, 1997, "Solution to Plato's Problem..." Psych. Review. Cognitive science perspective on learning word embeddings from text data. Experiments with an SVD-based embedding model, instead of the current neural-style logistic log-loss gradient learning approach.
• Reading: JM ch. 4, Logistic Regression and Classification
• Optional: Pang et al., EMNLP 2001 (supervised learning for sentiment classification)

Week 3: Neural Networks: Optimization, MLPs, RNNs

• 9/16: class
• 9/18: class
• Reading: JM ch. 6, Neural Networks
• Reading: JM ch. 13, Recurrent NNs
• Optional: Pascanu et al., ICML 2013, on vanishing gradients in RNNs

Other readings

Misc

• The Bitter Lesson (Sutton)

Attention

• There are multiple possible papers that could be called the first or most important for attention mechanisms (within NLP, at least).
  • Bahdanau et al., ICLR 2015 (paper that introduced attention)
  • Luong et al., EMNLP 2015
• [optional reading] An easy-to-read blog post on attention
• [optional watching] An good youtube video to understand self-attention

Transformers

• Cosma Shalizi notes in Transformers, including the kernel smoothing viewpoint (Tsai et al., EMNLP 2019) and many varied references
• Blogpost: Illustrated GPT-2
• OpenAI's GPT line of work consists of a series of non-peer-reviewed tech reports, with progressively less technical detail; for example, GPT (Radford et al. 2013), GPT-2 (Radford et al. 2019), GPT-3 (Brown et al. 2020), and InstructGPT (Ouyang et al. 2022), the last of which was used in the original December 2022 version of ChatGPT.