Homework assignments

Course topics outline

• Words and Classification Models
• Datasets and Classification Applications
• Linguistic Structure
• Embeddings and Neural Language Models
• Pre-trained LMs, BERT, generative LMs
• NLP Applications

Homeworks

• hw1.zip - released 9/18. For convenience: Notebook in HTML.
• hw2.pdf - released 10/5. Note Phase 1 and Phase 2 have separate due dates.
• hw3.pdf and cky.py - released 10/27. cky.py is the starter code.
• hw4 on google colab. (and ipynb file)

Schedule

Make sure to reload this page to ensure you’re seeing the latest version. Readings should be done before the indicated class. The main textbook is available free online:

• JM = Jurafsky and Martin, Speech and Language Processing, 3rd edition. (Main textbook.)
• INLP = Eisenstein, Intro to NLP, online text (or book version; we'll have readings from this one occasionally, and it's recommended in general.)

Tue 9/5: Introduction

• See the class survey exercise on Gradescope.

Thu 9/7: class cancelled

• Instructor Zoom office hours will be held instead.

Tue 9/12: Words and regular expressions

• Reading: JM ch. 2.1-2.4
• Optional: Ken Church, "Unix for Poets"
• Optional: RegexOne
• Optional: David C. Zentgraf, "What Every Programmer Absolutely, Positively Needs To Know About Encodings And Charater Sets To Work With Text"

Thu 9/14 N-gram language models (annotated)

• Reading: JM ch. 3.1-3.5
• Exercise #2: Predict the next word

Tue 9/19: Naive Bayes and Text Classification

• Reading: JM 4.1-4.6
• Optional: INLP 4.1-4.3
• In-class activity: Manual keywords. Keyword submission form, keyword classifier demo.
• 9/20 follow-up note: argmax and max operator notation

Thu 9/21: Logistic Regression Classification (annotated)

• Reading: JM ch. 5
• Exercise: Logistic regression example. Make-up version: create an example document with at least 4 features and a given weight vector. Calculate the probability of the y=1 class as per slide 12, showing your work. Submit on Gradescope.

Tue 9/26: Datasets and Annotation (annotated)

• Reading: INLP 4.5
• Exercise: Interannotator agreement (pdf)
• Lecture note on MLE

Thu 9/28: Classification Evaluation (annotated)

• Reading: JM 4.7-4.10 (don't need to know 4.9 yet)

Tue 10/3: Text Data Collection (annotated)

Thu 10/5: Final project overview

• Full details in the Final Project webpage

Tue 10/10: no class

Thu 10/12: Tagging (POS, NER) (annotated)

• Exercise: POS tags
• Reading: JM 8.1-8.3
• Reading: Justeson and Katz (1995)

Tue 10/17: Context-Free Grammars (CFG) (annotated)

• Reading: INLP 9.2. Other sections in Chapter 9 are optional but may be of interest.
• Exercise: English CFG

Thu 10/19: Constituency Parsing (annotated)

• Reading: INLP 10.1-10.2.
• Optional: JM ch. 17
• Exercise: CKY

Tue 10/24: Probabilistic Context-Free Grammars (PCFG) (annotated)

• Reading: INLP 10.3 (PCFGs) and 10.5 (lexicalized PCFGs)
• Lecture Zoom recording on Piazza resources page
• Don't forget - project proposal due tomorrow!

Thu 10/26: Dependency Syntax (annotated)

• Reading: JM 18.1-18.2 (Dependency relations, Transition-based parsers; can skip neural for now)

Tue 10/31: Word embeddings

• Reading: JM ch. 6, "Vector Semantics and Embeddings (for this week)

Thu 11/2: Word embeddings / Midterm review

• Word embeddings (part 2) slides
• Demo: exploring word embeddings
• Midterm practice problems and partial solutions

Tue 11/7: In-class midterm

Thu 11/9: No class

• Extra credit exercise: visit one of the poster sessions at TADA 2023 on either Thursday late afternoon, or Friday morning. Takes notes on a poster and write reflections with at least two paragraphs: one paragraph summarizing the work (include the main research question, what they did, and what they found), and one paragraph with your questions about the work or future work, and any other thoughts. Submit via Gradescope.

Tue 11/14: Neural networks in NLP

Thu 11/16: Transformers and BERT

Tue 11/21: Statistics: Is my experimental result just a fluke?

• Reading: INLP 4.4.3
• Video: Solve Every Statistics Problem with One Weird Trick (on the bootstrap; Jonathan Stray, 2016)
• Demos: randomized null hypothesis testing, bootstrap confidence intervals
• Project progress report due (see Gradescope)

Thu 11/23: no class

Tue 11/28, Thu 11/30: Generative Large Language Models

• Blog post: The Illustrated GPT-2 (Alammar, 2018)
• T5 (Raffel et al., JMLR 2020)
• Nucleus sampling (Holtzman et al., ICLR 2020)
• The GPT* line of work are non-peer-reviewed tech reports, with progressively less and less technical detail; for example, GPT-2 (Radford et al. 2019), GPT-3 (Brown et al. 2020), and instruction tuning thereof (Ouyang et al. 2022), the last of which was used in ChatGPT; to keep the various GPT variants straight, consider the Wikipedia page.
• Recent (2023) prompt engineering resources: Prompt Engineering Guide (promptingguide.ai), Prompt Engineering (Lilian Weng)
• On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (Bender et al., FAccT 2021)

Note HW4 is due this weekend (link on top of page).

Tue 12/5, Thu 12/7: Final presentations

• Project presentations will be in class this week. See instructions in the Google Sheets slidedecks.
• Day 1 program, Day 2 program
• Exercise, due Friday: project presentation writeup

Final project reports are due by the last day of the final exam period (Dec 15; see project page). Late days cannot be used.