### Learning Representation
And Behavior: Manifold and Spectral Methods for Markov
Decision Processes and Reinforcement Learning

### A Tutorial to be given
at ICML 2006

### Carnegie Mellon
University, Sunday June 25, 2006

### SUMMARY

This tutorial is an interdisciplinary presentation on a novel
framework for solving Markov decision processes and reinforcement
learning problems, using multiscale spectral and manifold learning
methods. Manifold learning methods are an exciting new framework for
machine learning. Previous work on manifold and spectral methods have
largely focused on dimensionality reduction, (semi-)supervised
learning and clustering. Furthermore, manifold and spectral techniques
have mostly focused on Laplacian or Fourier-based global approaches.
This tutorial surveys new emerging connections between research in
manifold learning and Markov decision processes and reinforcement
learning. The tutorial will also introduce * diffusion wavelets
*, a novel class of multiresolution wavelet-based manifold learning
methods which are not well known in the machine learning
community. Together, the Laplacian and wavelet based manifold learning
methods hold the promise of a new generation of powerful tools for
solving MDPs and RL, including ways of approximating value functions
that respect geodesic distances on the underlying manifold; faster
methods of policy evaluation and novel variants of policy iteration
where both the representation and optimal policy can be simultaneously
learned; algorithms for hierarchical reinforcement learning where the
underlying hierarchy is automatically learned; novel approaches to
transfer learning by transferring shared representations; and enabling
reinforcement learning methods without requiring (task-specific)
rewards.

The tutorial will be accesssible to researchers and graduate
students working in any area of machine learning or related areas
(robotics, statistics etc.). The tutorial will include a detailed
introduction to the underlying mathematics as well as describe basis
construction algorithms and how these lead to novel ways of solving
MDPs and RL problems. All the ideas will be illustrated using hands-on
MATLAB demonstrations.

### Tutorial Slides

Tutorial slides.