Advanced Topics in Natural Language Processing

CS 685, Spring 2021, UMass Amherst

Instructor: Brendan O'Connor
TA: Nader Akoury
Lectures: MW 2:30PM - 3:45PM, online on Zoom
Office Hours (US Eastern Time), see Slack #general channel for meeting links

• Nader: Thu 10-11AM
• Brendan: Tues 9:30-10:30AM

Links:

• Schedule
• Problem Sets
• Final Project
• Literature Review and Reading Reviews
• Feedback form
• Grading and Polcies
• Slack for communication
• Gradescope for turning in assignments
• Zoom for lectures: see link on top of Slack #general channel

Please contact Nader if you are having issues accessing any of the resources.

Course Information

Description: This course covers a broad range of advanced level topics in natural language processing. It is intended for graduate students in computer science who have familiarity with machine learning fundamentals, and previous course or research experience in natural language processing. It may also be appropriate for computationally sophisticated students in linguistics and related areas. Topics include probabilistic models of language, computationally tractable linguistic representations for syntax and semantics, neural network models for language, and selected topics in discourse and text mining. After completing the course, students should be able to read and evaluate current NLP research papers. Coursework includes a research literature review, homework assignments, and a final project.

Prerequisites: Familiarity with multivariate calculus, probability theory, dynamic programming, and mathematical foundations of machine learning algorithms, such as from COMPSCI 688 (Probabilistic Graphical Models), 689 (Machine Learning), or equivalent. COMPSCI 682 (Deep Learning) will be helpful, but not required. Experience in linguistics will also be helpful, but not required.

Course topics

CS685 is an Advanced Topics course, with different focuses each semester. This semester, we will focus on

• Machine learning fundamentals for NLP.
• "Syn-Sem-Soc" linguistics in NLP:
  1. Syntax: the structure of language;
  2. Semantics: the meaning of language; and
  3. Social: the human behavior and interactions behind language.
• Deep learning in NLP, understood through all of the above.

Related courses

See also previous versions or variants of this course: 685 in Fall 2020, 685 in Spring 2020, 690D in Spring 2019, and 690N in Spring 2018. See also this list of courses in NLP and related areas offered at UMass and nearby.

Readings

We'll use the Eisenstein text for many of the readings in this course.

In general there are a number of useful texts on NLP, as well as important adjacent areas like linguistics and machine learning. For some of the links below, you need to sign in to the UMass VPN (or access from campus) to get access via the UMass Library subscription.

• Eisenstein (2019), Introduction to Natural Language Processing, with a free PDF version ("INLP").
• Jurafsky and Martin (~2021), Speech and Language Processing (3rd ed) ("SLP").
• Two short books briefly overviewing linguistics topics relevant to NLP (use campus VPN to access), in the areas of:
  • Morphology and Syntax: Bender (2013), Linguistic Fundamentals for NLP.
  • Semantics and Pragmatics: Bender and Lascarides (2019), Linguistic Fundamentals for NLP II.
• Manning and Schütze, Foundations of Statistical NLP. Free access at UMass.
• Murphy, Machine Learning: a Probabilistic Perspective (a few different editions/volumes there).
• Goodfellow, Bengio and Courville (2016), Deep Learning.
• Bird et al, NLP with Python, a.k.a. the NLTK book. Free and very introductory (some folks use it without any programming background), with a computational linguistics emphasis.

Note some of these authors were recently lamenting the lack of sociolinguistics in the usual NLP curriculum, and its lack of presence in their own books! There is much to learn and intellectual change in progress.

Math review

Calculus, linear algebra, and probability theory are the required mathematical background for the course. For review, especially of the HW0 topics, the following resources may be helpful.

• Appendix A (probability) and Appendix B (optimization) in Eisenstein (2019).
• Stanford CS229 Linear Algebra review/reference.
• I've heard some students who like MathematicalMonk's probability videos.
• Wikipedia is usually a good reference for mathematical topics. (For specific NLP or ML topics, like, say, evaluation metrics, it can be trickier since sometimes it's written in a subdiscipline-specific way. More basic mathematical topics, however, tend to have broader consensus on terminology and concepts.)

Courses elsewehre