CS 690S: AI Alignment

Schedule

Tue 09/08 — Course overview [Slides]

• Optional: The Alignment Problem from a Deep Learning Perspective

Thu 09/10 — Behavior cloning [Slides]

• Required: Behavioral Cloning from Observation


• Optional: π_0: A Vision-Language-Action Flow Model for General Robot Control


• Assignment 1 — Due by 11:59pm on Thurs 9/24 via Gradescope.

Tue 09/15 — Limitations of behavior cloning [Slides]

• Required: A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning


• Optional: Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap

Thu 09/17 — Reinforcement Learning [Slides]

• Required: Introduction (Part 1 and Part 2) and Algorithm Docs (Proximal Policy Optimization): Spinning Up in Deep RL


• Optional for more background: Sutton/Barto RL Book


• Optional: Proximal Policy Optimization Algorithms


• Optional: Soft Actor-Critic

Tue 09/22 — Reward hacking [Slides]

• Required: Specification gaming: the flip side of AI ingenuity


• Required: Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals


• Optional: The Perils of Trial-and-Error Reward Design: Misdesign through Overfitting and Invalid Task Specifications


• Optional: Consequences of Misaligned AI

Thu 09/24 — Inverse reinforcement learning [Slides]

• Required: Apprenticeship Learning via Inverse Reinforcement Learning


• Optional: A Game-Theoretic Approach to Apprenticeship Learning

Tue 09/29 — Maximum entropy inverse reinforcement learning [Slides]

• Required: Maximum Entropy Inverse Reinforcement Learning


• Optional: Maximum Entropy Deep Inverse Reinforcement Learning


• Assignment 2 — Due by 11:59pm on Thurs 10/13 via Gradescope.

Thu 10/01 — Bayesian inverse reinforcement learning [Slides]

• Required: Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning


• Optional: Bayesian Inverse Reinforcement Learning

Tue 10/06 — Occupancy matching [Slides]

• Required: Generative Adversarial Imitation Learning


• Optional: Dual RL: Unification and New Methods for Reinforcement and Imitation Learning


• Final project proposal — Due by 11:59pm on Thurs 10/20 via Gradescope.

Thu 10/08 — Interactive RL [Slides]

• Required: Interactively shaping agents via human reinforcement: The TAMER framework


• Optional: Interactive Learning from Policy-Dependent Human Feedback


• Optional: Efficiently Guiding Imitation Learning Agents with Human Gaze


• Optional: The EMPATHIC Framework for Task Learning from Implicit Human Feedback

Tue 10/13 — In-class midterm (covers material up to and including Interactive RL)




Thu 10/15 — Reinforcement learning from human feedback: indirect methods [Slides]

• Required: Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations


• Optional: Training language models to follow instructions with human feedback


• Optional: Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback


• Assignment 3 — Due by 11:59pm on Thurs 10/29 via Gradescope.

Tue 10/20 — Models of human preference [Slides]

• Required: Models of Human Preference for Learning Reward Functions


• Optional: Human Irrationality: Both Bad and Good for Reward Inference

Thu 10/22 — Reinforcement learning from human feedback: direct methods [Slides]

• Required: Direct Preference Optimization: Your Language Model is Secretly a Reward Model


• Optional: Contrastive Preference Learning: Learning from Human Feedback without RL

Tue 10/27 — Performance Guarantees for RLHF [Slides]

• Required: Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences


• Optional: Value Alignment Verification

Thu 10/29 — Reward design and inference from language [Slides]

• Required: Eureka: Human-Level Reward Design via Coding Large Language Models


• Optional: RL Zero: Zero-Shot Language to Behaviors Without Any Supervision


• Final project check-in — Due by 11:59pm on Thurs 11/12 via Gradescope.

Tue 11/03 — Election day




Thu 11/05 — Cooperation and Assistance Games [Slides]

• Required: Cooperative Inverse Reinforcement Learning


• Optional: AssistanceZero: Scalably Solving Assistance Games

Tue 11/10 — AI risks [Slides]

• Required: Unsolved Problems in AI Safety


• Optional: Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

Thu 11/12 — AI risk case studies [Slides]

• Required: OpenAI Hugging Face Incident Report


• Optional: AISI report

Tue 11/17 — Scalable oversight [Slides]

• Required: Measuring Progress on Scalable Oversight for Large Language Models


• Optional: Constitutional AI: Harmlessness from AI Feedback


• Optional: Let's Verify Step by Step


• Optional: Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Thu 11/19 — Mechanistic interpretability [Slides]

• Required: Mechanistic Interpretability for AI Safety (Section 4 only)


• Optional: Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models for Mechanistic Interpretability

Tue 11/24 — Thanksgiving break




Thu 11/26 — Thanksgiving break




Tue 12/01 — Red teaming [Slides]

• Required: Red Teaming Language Models with Language Models


• Optional: Explore, Establish, Exploit: Red Teaming Language Models from Scratch


• Final project report — Due by 11:59pm on Tuesday 12/15 via Gradescope.

Thu 12/03 — Adversarial robustness [Slides]

• Required: Universal and Transferable Adversarial Attacks on Aligned Language Models


• Optional: Fundamental Limitations of Alignment in Large Language Models

Tue 12/08 — Frontiers of misalignment [Slides]

• Required: Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs


• Optional: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training


• Optional: Evaluating Frontier Models for Stealth and Situational Awareness


• Optional: Sycophancy to subterfuge: Investigating reward-tampering in large language models

Thu 12/10 — Final Project Presentations




Tue 12/15 — Final Project Presentations

Grading

Grades will be calculated as follows, using a scale that includes both plus and minus letter grades:

• 15% Reading critiques:
  
  A written critique of the required reading for each class will be due by 11:59 PM the previous night via Gradescope. Each critique should be formatted as a numbered list that addresses all of the following:
  
  
  • (1) A short summary of the main contribution(s) of the paper in your own words (roughly two sentences)
  • (2) A short description of how the paper differs from prior work.
  • (3) One strength and one weakness of the proposed method, core argument, or experiments
  • (4) At least one question / comment that you'd like me to address during class or that could spur discussion
  • (5) One idea for potential follow-up work, or a citation and short description of existing follow-up work that cites this paper.
  
  In all cases, the written critique should provide non-trivial insight into the reading. To get full credit, you must show that you understood and thought critically about the core concepts presented. Students are strongly urged to not use AI to assist in writing these, as they provide an opportunity to prepare for the exams and to fuel in-class discussion.
  
  
  
• 15% Homework assignments:
  
  There will be several programming assignments, in which machine learning algorithms will be implemented and evaluated. Each assignment will require students to turn in code as well as a short written report.
  
  
  
• 40% Midterm and final exams:
  
  Since generative AI is permitted to be used for every other part of the course, there will an midterm (in-class) and final (during finals week) that tests core knowledge about the readings and homeworks. It will be designed such that you should have all the knowledge you need to do well as long as you attended class and did not over-rely on AI to complete the reading responses and homeworks.
  
  
  
• 30% Final project:
  
  Roughly halfway through the semester, students will propose topics of their own choosing for a large final programming project. These projects may be completed alone, though it is encouraged to work in groups of up to 3 students. A rough guideline is that the project produce about half a standard conference paper worth of material (this means both technical content and length—about 4 double-column pages in LaTeX).
  
  These projects are a chance to dive deeply into any topic of interest related to the course. Students are encouraged to tie this work into their primary research that they are already pursuing, as long as it can relate to AI alignment broadly).
  
  Example projects could include extending an algorithm in a novel way, comparing several algorithms on an interesting problem, or designing a new approach to attack a problem relevant to the class. In all cases, there should be a novel intellectual contribution, as well as empirical results on a problem of interest.
  
  
  

Late work policy: Reading critiques will not be accepted late, since their main goal is to provide fuel for discussion. However, each student can skip up to three critiques without penalty. No other extensions will be given for critiques except in highly unusual circumstances, so please save these for times of necessity.

All other assignments can be turned in up to one week late, at a loss of 2 points (out of 100) per late day (though this cannot go beyond the final day of classes). However, please note that the midterm may include material from assignments shortly after their due date, so completing them on time is strongly encouraged.

Attendance policy: Given that discussion is an important element of the course, attendance is mandatory.

Grades will be assigned using both plus and minus grades as follows:

93-100: A
90-93: A-
87-90: B+
83-87: B
80-83: B-
77-80: C+
73-77: C
<73: F

Generative AI Policy

Generative AI tools can be used for any aspect of the class, other than the midterm and final. However, it is strongly recommended that you use them sparingly and with a great deal of review, to ensure that you learn the material sufficiently well. This is both for your own learning benefit, as well as to ensure that you perform well on the midterm that covers this material. Aside from the use of generative AI, all assignments must be completed alone unless otherwise specified.

Academic Integrity Statement

UMass Amherst is strongly committed to academic integrity, which is defined as completing all academic work without cheating, lying, stealing, or receiving unauthorized assistance from any other person, or using any source of information not appropriately authorized or attributed. As a community, we hold each other accountable and support each other’s knowledge and understanding of academic integrity. Academic dishonesty is prohibited in all programs of the University and includes but is not limited to: Cheating, fabrication, plagiarism, lying, and facilitating dishonesty, via analogue and digital means. Sanctions may be imposed on any student who has committed or participated in an academic integrity infraction. Any person who has reason to believe that a student has committed an academic integrity infraction should bring such information to the attention of the appropriate course instructor as soon as possible. All students at the University of Massachusetts Amherst have read and acknowledged the Commitment to Academic Integrity and are knowingly responsible for completing all work with integrity and in accordance with the policy: (https://www.umass.edu/senate/book/academic-regulations-academic-integrity-policy)

In this course, you are encouraged to discuss the readings and concepts with classmates, but all work must be your own. Programming assignments and homework must be completed alone, except for teams for the final project. Generative AI may be used in accordance with the policy listed above.

Accommodation Statement

The University of Massachusetts Amherst is committed to making reasonable, effective, and appropriate accommodations to meet the needs of students with disabilities and help create a barrier-free campus. If you have a disability and require accommodations, please register with Disability Services, meet with an Access Coordinator in Disability Services, and send your accommodation letter to your faculty. Information on services and registration is available on the Disability Services website (https://www.umass.edu/disability/)

Title IX Statement

In accordance with Title IX of the Education Amendments of 1972 that prohibits gender-based discrimination in educational settings that receive federal funds, the University of Massachusetts Amherst is committed to providing a safe learning environment for all students, free from all forms of discrimination, including sexual assault, sexual harassment, domestic violence, dating violence, stalking, and retaliation. This includes interactions in person or online through digital platforms and social media. Title IX also protects against discrimination on the basis of pregnancy, childbirth, false pregnancy, miscarriage, abortion, or related conditions, including recovery. There are resources here on campus to support you. A summary of the available Title IX resources (confidential and non-confidential) can be found at the following link: https://www.umass.edu/titleix/resources. You do not need to make a formal report to access them. If you need immediate support, you are not alone. Free and confidential support is available 24 hours a day / 7 days a week / 365 days a year at the SASA Hotline 413-545-0800.