Learning Representation And Behavior: Manifold and Spectral Methods for Markov Decision Processes and Reinforcement Learning

A Tutorial to be given at ICML 2006

Carnegie Mellon University, Sunday June 25, 2006

Sridhar Mahadevan (U.Mass, Amherst) Mauro Maggioni (Yale University)


This tutorial is an interdisciplinary presentation on a novel framework for solving Markov decision processes and reinforcement learning problems, using multiscale spectral and manifold learning methods. Manifold learning methods are an exciting new framework for machine learning. Previous work on manifold and spectral methods have largely focused on dimensionality reduction, (semi-)supervised learning and clustering. Furthermore, manifold and spectral techniques have mostly focused on Laplacian or Fourier-based global approaches.

This tutorial surveys new emerging connections between research in manifold learning and Markov decision processes and reinforcement learning. The tutorial will also introduce diffusion wavelets , a novel class of multiresolution wavelet-based manifold learning methods which are not well known in the machine learning community. Together, the Laplacian and wavelet based manifold learning methods hold the promise of a new generation of powerful tools for solving MDPs and RL, including ways of approximating value functions that respect geodesic distances on the underlying manifold; faster methods of policy evaluation and novel variants of policy iteration where both the representation and optimal policy can be simultaneously learned; algorithms for hierarchical reinforcement learning where the underlying hierarchy is automatically learned; novel approaches to transfer learning by transferring shared representations; and enabling reinforcement learning methods without requiring (task-specific) rewards.

The tutorial will be accesssible to researchers and graduate students working in any area of machine learning or related areas (robotics, statistics etc.). The tutorial will include a detailed introduction to the underlying mathematics as well as describe basis construction algorithms and how these lead to novel ways of solving MDPs and RL problems. All the ideas will be illustrated using hands-on MATLAB demonstrations.

Tutorial Slides

  • Tutorial slides.