COMPSCI 687: Reinforcement Learning

Course Overview

This course provides an introduction to reinforcement learning. Reinforcement learning algorithms repeatedly answer the question “What should be done next?”, and they can learn through trial and error to answer these questions even when there is no supervisor specifying the correct answer. Applications span medicine, marketing, robotics, game playing, environmental applications, dialogue systems, and many other areas.

Broad topics include Markov decision processes; model-based and model-free methods; batch and online methods; value-function methods; actor-critic and policy-gradient methods; and representations for reinforcement learning. Special topics may include safe reinforcement learning, hierarchical reinforcement learning, reinforcement learning theory, multi-agent reinforcement learning, fine-tuning large language models, and connections to animal learning.

Prerequisites and Expected Background

The course has no formally enforced prerequisite courses. It assumes a very strong background in calculus, linear algebra, and strategies for proving theorems; strong programming ability; background in machine learning comparable to COMPSCI 589 or COMPSCI 689; and background in artificial intelligence comparable to COMPSCI 683. The course is intended primarily for graduate Computer Science students.

Student Learning Objectives

By the end of the course, students should be able to:

• formulate reinforcement learning problems as partially observable Markov decision processes and apply standard algorithms to solve them;
• explain the main ideas behind foundational reinforcement learning algorithms and concepts; and
• distinguish among broad classes of reinforcement learning methods and describe how reinforcement learning methods are used to fine-tune large language models.

People

Instructor: Professor Philip S. Thomas (pthomas@umass.edu)

Teaching assistants: Norman Zhang (renhaozhang@umass.edu), Shauna Choi (seohyunchoi@umass.edu), and Shruti Chanumolu (schanumolu@umass.edu)

Office hours:

• Monday: 10–12, Shruti Chanumolu, CSL E211
• Tuesday: 10–12, Shauna Choi, CSL E211
• Wednesday: 2–4, Norman Zhang, CSL E213

Important Dates

• First Test: October 14, 2026, 7–9 p.m., Integrative Learning Center (ILC) Room N151
• Second Test: Date to be determined, around November 23, 2026
• Final Exam: December 21, 2026, 8 a.m.–12 p.m., CSL E110

Syllabus

The syllabus contains the tentative schedule, grading scheme, submission and make-up policies, and other detailed course policies.

Download the Fall 2026 syllabus

Course Notes

Download the current course notes

Assignments

Homework 1 (PDF) · LaTeX source
Assigned: September 9, 2026
Due: September 16, 2026, at midnight (end of day, Eastern time).

Quizzes

Pop Quiz 1 (PDF) · Solution (PDF)
Date: TBD, likely September 16, 2026.

Course Communication and Materials

There is no required textbook. Typed notes will be provided on this page. Canvas will contain the syllabus and links to this course page and Gradescope, but it will not otherwise be used for course content. Course announcements will be sent to students’ UMass email addresses.

Lecture recordings and practice tests will not be provided. See the syllabus for the complete attendance, grading, collaboration, generative-AI, and academic-integrity policies.