Homework assignments

Course topics, approximate outline

• Text Classification
• Linguistic Structure
• Neural Language Models

Homeworks

• hw1.zip - released 2/9, due 2/21. For convenience: Notebook in HTML.
• hw2.pdf - released 3/1. See doc for due dates.
• hw3.pdf and cky.py - released 3/30, due 4/8.
• hw4 (colab), due 5/10.

Schedule

Lecture topics will appear below. Make sure to reload this page to ensure you’re seeing the latest version. Readings should be done before the indicated class. The main textbook is available free online:

• JM = Jurafsky and Martin, Speech and Language Processing, 3rd edition. (Main textbook.)
• INLP = Eisenstein, Intro to NLP, online text (or book version; we'll have readings from this one occasionally, and it's recommended in general.)

Thu 2/1: Introduction

Tue 2/6: Words and regular expressions

• Reading: JM ch. 2.1-2.4
• Optional: Ken Church, "Unix for Poets"
• Optional: RegexOne
• Optional: David C. Zentgraf, "What Every Programmer Absolutely, Positively Needs To Know About Encodings And Charater Sets To Work With Text"

Thu 2/8 (+2/13): Text Classification and Naive Bayes (annotated)

• Reading: JM 4.1-4.6
• Optional: INLP 4.1-4.3
• In-class activity: Manual keywords. keyword classifier demo (ipynb)

Tue 2/13 (+2/15): Logistic Regression Classification (annotated)

• Reading: JM ch. 5.1-5.5

Thu 2/15: Logistic Regression, continued

• Using slides above

Tue 2/20: Classification Evaluation and Generalization

• Reading: JM 5.7
• Reading: JM 4.7, 4.8, 4.10
• Exercise 2: Logistic regression - if you turned it in on paper with your name, you're done! If you missed class, please submit on Gradescope by this Friday.

Thu 2/22: no class (Monday class schedule day)

Tue 2/27: Annotations

• Reading: INLP, 4.5
• Exercise: Interannotator agreement

Thu 2/29: Text Data and Final Project Overview

• Full project details in the Final Project webpage

Tue 3/5: Tagging (annotated)

• Exercise: POS tags
• Reading: JM 8.1-8.3
• Reading: Justeson and Katz (1995)

Thu 3/7: Context-Free Grammars (annotated)

• Reading: INLP 9.2. Other sections in Chapter 9 are optional but may be of interest.
• Exercise: English CFG

Tue 3/12: Parsing (CKY algorithm)

• Reading: INLP 10.1-10.2.
• Optional: JM ch. 17
• Exercise: CKY

Thu 3/14: Dependencies and project work session

• Also reviewed the CKY exercise and CKY runtime
• Reading: JM 18.1 (Dependency relations)

Tue 3/19, Thu 3/21: no class (Spring break)

Tue 3/26: N-Gram Language Models

• Reading: JM ch. 3.1-3.5
• Exercise: Predict the next word

Thu 3/28: Word Embeddings (I)

• Reading: JM ch. 6, "Vector Semantics and Embeddings (this session and next week)

Tue 4/2: Class cancelled (sickness)

Thu 4/4: Word Embeddings (II)

• Readings: Garten et al. (DDR) and Demszky et al. (clustering) from the slides. Only for high level concepts as discussed in lecture.

Tue 4/9: Midterm review

• Practice midterm questions / topics

Thu 4/11: In-class midterm

Tue 4/16: Neural networks

Thu 4/18: Transformers and BERT

• JM ch. 10.1-10.6 on Transformers
• JM ch. 11 on BERT (MLM + fine-tuning)
• Rogers et al. 2020, "A Primer in BERTology: What We Know About How BERT Works", TACL
• Manning et al. 2020, "Emergent linguistic structure in artificial neural networks trained by self-supervision", PNAS

Tue 4/23: BERT demo

• In-class Colab demo for BERT and MLM

Thu 4/25: Generative LLMs (incl. GPT)

• JM ch. 10.7-10.8 on Transformer LMs
• Blog post: The Illustrated GPT-2 (Alammar, 2018)
• T5 (Raffel et al., JMLR 2020)
• Nucleus sampling (Holtzman et al., ICLR 2020)
• The GPT* line of work are non-peer-reviewed tech reports, with progressively less and less technical detail; for example, GPT-2 (Radford et al. 2019), GPT-3 (Brown et al. 2020), and instruction tuning thereof (Ouyang et al. 2022), the last of which was used in the original December 2022 release of ChatGPT. To keep the subsequence (many) GPT variants straight, consider the Wikipedia page.
• Some prompt engineering resources (a very rapidly changing issue): Prompt Engineering Guide (promptingguide.ai), Prompt Engineering (Lilian Weng)

Tue 4/30, Thu 5/2: Challenges and Issues for LLMs

• Bender et al., FAccT 2021: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
• Dahl et al., 2024: Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
• Min et al., EMNLP 2023: FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

Tue 5/7, Thu 5/9: Final presentations

5/17: last day of semester; final projects due.