Advanced Natural Language Processing

CS 685, Spring 2025, UMass Amherst CS
Mon/Wed 2:30-3:45 PM in Goessman 64
This year, this class won't be recorded as previous years. If you want to see Mohit's recording of the course last year, please go to here

Instructor: Haw-Shiuan Chang (Most of the materials are prepared by Mohit Iyyer)
TAs: TBA
Email (to all of us): cics.685.instructors@gmail.com

• (The response of emails sent to this address might be delayed. If you have questions about homework or course related materials, please ask them using Piazza)
• (Please send email to the instructors if you have unforeseen health and personal emergencies that would affect your homework submission.)

• TBA

• Schedule
• Grading / Policies
• Gradescope (for assignment submission)
• Canvas (only for final grades)
• Piazza (for questions and discussion)

Course description

Natural Language Processing (NLP) is the engineering art and science of how to teach computers to understand human language. NLP is a type of artificial intelligence technology, and it's now ubiquitous -- NLP lets us talk to our phones, use the web to answer questions, map out discussions in books and social media, and even translate between human languages. Since language is rich, ambiguous, and very difficult for computers to understand, these systems can sometimes seem like magic -- but these are engineering problems we can tackle with data, math, and insights from linguistics.

This course will broadly deal with deep learning methods for natural language processing, with a specific focus on large language models. Most of the semester will focus on neural language models. It is intended for graduate students in computer science and linguistics who are (1) interested in learning about cutting-edge research progress in NLP and (2) familiar with machine learning fundamentals. We will cover modeling architectures, training objectives, and downstream tasks (e.g., text classification, question answering, and text generation). Coursework includes understanding the course materials, programming assignments, and a final project. This is an in-person class.

Readings

A nice textbook for NLP fundamentals is Jurafsky and Martin, Speech and Language Processing, 3rd ed. For this course, readings will mainly be NLP conference papers (e.g., from ACL, NAACL, and EMNLP). We will post all readings as PDFs.

Other useful texts for NLP include:

• Manning and Schütze, Foundations of Stat NLP. Free access at UMass.
• Eisenstein, Natural Language Processing. Draft textbook.
• Smith, Linguistic Structure Prediction. Free access at UMass. Short book. Excellent coverage of structured prediction inference methods for NLP.
• Murphy, Machine Learning: a Probabilistic Perspective. Excellent, though advanced, coverage of most of the machine learning methods we will use.
• Bender, Linguistic Fundamentals for NLP. Short book. Focuses on linguisic issues relevant to NLP.
• Bird et al, NLP with Python, a.k.a. the NLTK book. Aimed at a more introductory level than this course, but the book is a good gentle introduction to NLP with a CL (computational linguistics) emphasis. The NLTK software has easy-to-use data access and some interfaces to (not always SOTA) NLP tools.