Advanced Natural Language Processing

CS 685, Spring 2024, UMass Amherst CS
Mon/Wed 2:30-3:45 PM in Goessman 64
This class will also be livestreamed on Youtube! See schedule for all videos, readings, and assignments.

Instructor: Mohit Iyyer
TAs: Chau Pham, Yekyung Kim, Katherine Thai (remote), Saurabh Bajaj
Email (to all of us): cs685instructors@gmail.com
Office hours (US Eastern time), see Piazza for meeting links:

• Katherine: Mon 10-11am (remote only)
• Yekyung: Tues 2-3pm, CS207 Cube 3
• Mohit: Wed 4-5pm, CS232
• Chau: Thu 1-2pm, CS207 Cube 1
• Saurabh: Fri 11am-12pm, CS207 Cube 3

• Schedule
• Grading / Policies
• Anonymous question/comment form
• Gradescope (for assignment submission)
• Moodle (only for final grades)
• Piazza (for discussion)

Course description

Natural Language Processing (NLP) is the engineering art and science of how to teach computers to understand human language. NLP is a type of artificial intelligence technology, and it's now ubiquitous -- NLP lets us talk to our phones, use the web to answer questions, map out discussions in books and social media, and even translate between human languages. Since language is rich, ambiguous, and very difficult for computers to understand, these systems can sometimes seem like magic -- but these are engineering problems we can tackle with data, math, and insights from linguistics.

This course will broadly deal with deep learning methods for natural language processing, with a specific focus on large language models. Most of the semester will focus on neural language models and transfer learning, both of which have significantly pushed forward the state of the art. It is intended for graduate students in computer science and linguistics who are (1) interested in learning about cutting-edge research progress in NLP and (2) familiar with machine learning fundamentals. We will cover modeling architectures, training objectives, and downstream tasks (e.g., text classification, question answering, and text generation). Coursework includes reading recent research papers, programming assignments, and a final project. While this is an in-person class, all lectures will be livestreamed on YouTube, and video links will be posted on the course schedule.

Readings

A nice textbook for NLP fundamentals is Jurafsky and Martin, Speech and Language Processing, 3rd ed. For this course, readings will mainly be NLP conference papers (e.g., from ACL, NAACL, and EMNLP). We will post all readings as PDFs.

Other useful texts for NLP include:

• Manning and Schütze, Foundations of Stat NLP. Free access at UMass.
• Eisenstein, Natural Language Processing. Draft textbook.
• Smith, Linguistic Structure Prediction. Free access at UMass. Short book. Excellent coverage of structured prediction inference methods for NLP.
• Murphy, Machine Learning: a Probabilistic Perspective. Excellent, though advanced, coverage of most of the machine learning methods we will use.
• Bender, Linguistic Fundamentals for NLP. Short book. Focuses on linguisic issues relevant to NLP.
• Bird et al, NLP with Python, a.k.a. the NLTK book. Aimed at a more introductory level than this course, but the book is a good gentle introduction to NLP with a CL (computational linguistics) emphasis. The NLTK software has easy-to-use data access and some interfaces to (not always SOTA) NLP tools.