Advanced Natural Language Processing

COMPSCI 685, Fall 2025, UMass Amherst CICS
Tue/Thu 4-5:15pm in Herter Hall, room 231

Instructor: Brendan O'Connor
TAs: Nguyen Tran, Rohan Pandey, Marisa Hudspeth
Email (reaches instructor and TAs): cics.685.instructors@gmail.com

• If you have questions about homework or course related materials, please ask them using Piazza.
• Use this email address for other matters, including extensions regarding health or personal emergencies.

Office hours, starting week of Sept. 8:

• Marisa: Tuesdays 10–11 AM, in room CS207, Cube 1
• Rohan: Wednesdays 2:30–3:30 PM, in room CS207, Cube 1
• Brendan: Thursdays 9:30-10:30 AM, in office CS238. (No OH 9/25.)
• Nguyen: currently no office hours (will start in November)
• Note: Times are US Eastern Time. Some sessions may be hybrid with the main zoom link (below). For latest scheduling updates, see Piazza.

Links:

• Schedule
• Grading / Policies
• Gradescope for assignment submission.
• Piazza for questions and discussion. Ask instructors email for code to join, if needed.
• Zoom link for hybrid OH or occasional online sessions; requires Five College login.
• Canvas/Echo for lecture video recordings.

Course description

Natural Language Processing (NLP) is the engineering art and science of how to teach computers to understand human language. NLP is a type of artificial intelligence technology, and it's now ubiquitous -- NLP lets us talk to our phones, use the web to answer questions, map out discussions in books and social media, and even translate between human languages. Since language is rich, ambiguous, and very difficult for computers to understand, these systems can sometimes seem like magic -- but these are engineering problems we can tackle with data, math, and insights from linguistics.

This course will broadly deal with deep learning methods for natural language processing, with a specific focus on large language models and neural language models. It is intended for graduate students in computer science and linguistics who are (1) interested in learning about cutting-edge research progress in NLP and (2) familiar with machine learning fundamentals. We will cover modeling architectures, training objectives, and downstream tasks (e.g., text classification, question answering, and text generation). Coursework includes understanding the course materials, programming assignments, and a final project. This is an in-person class.

Prerequisites: This course assumes mathematical preparation appropriate for graduate-level machine learning coursework, including linear algebra, multivariate calculus, and probability theory. Since students come from different institutions with different types of preparation, we do not have formal prerequisites; depending on preparation, students may need more or less time to review relevant mathematical background. Note that UMass offers a wide array of other NLP and NLP-related courses that may also be of interest; for example, COMPSCI 485, LING 492B, and DACSS 758 typically assume less mathematical or machine learning background on certain topics. See other UMass NLP courses listed here.

Readings

A nice textbook for NLP fundamentals is Jurafsky and Martin, Speech and Language Processing, 3rd ed. We will draw readings from their online August 2025 edition, as well as NLP conference papers (e.g., from ACL, NAACL, and EMNLP). We will post all readings as PDFs.

Some other textbooks on NLP and related topics include:

• Eisenstein 2018, Natural Language Processing (draft, book webpage). Probably the best text broadly covering pre-Transformer/LLM NLP topics.
• Goodfellow et al. 2016, Deep Learning. Excellent coverage of pre-Transformers neural network methods and architectures.
• MacKay 2003, Information Theory, Inference, and Learning Algorithms. The most fun and interesting approach to ML/stat/infotheory material related to NLP. Too idiosyncratic to be a general ML reference, but more inspirational.
• A general ML text is important to learn or use as reference; these two are good, and preference between them is a matter of taste:
  • Murphy, Machine Learning: a Probabilistic Perspective.
  • Bishop, Pattern Recognition and Machine Learning.
• Manning and Schütze, Foundations of Statistical NLP; much older than others on this list. Free access at UMass. Lillian Lee wrote an interesting review (2000).
• Smith, Linguistic Structure Prediction. Free access at UMass. Short book. Focused on structured prediction inference methods for NLP.
• Bird et al, NLP with Python, a.k.a. the NLTK book. Aimed at a more introductory level than this course, but the book is a good gentle introduction to NLP with a CL (computational linguistics) emphasis.