Machine Learning and Friends Lunch
SAMUEL MEETS AMAREL: A Coordinate-Free Framework for Learning Behavior and Representation
Sridhar Mahadevan
UMass
Abstract
The American Association for Artificial Intelligence marks its 25th
anniversary this year. It is an opportune moment to celebrate the
legacy of the past, and envision research directions that may shape
the field over the next quarter century. With this overarching goal in
mind, I describe my recent work on a longstanding intellectual puzzle:
how can agents discover representations from their experience to guide
sequential decision-making?
I present a mathematical framework unifying Samuel and Amarel's five
decade old paradigms of behavior and representation learning. The
proposed framework exploits intricate connections between continuous
and discrete mathematics: Riemannian manifolds and the spectral theory
of graphs; elliptic differential equations and abstract harmonic
analysis; self-adjoint operators on Hilbert spaces and functions on
graphs. To illustrate the framework, I describe a novel approach to
developmental learning where agents learn "proto-value" functions from
their experience. Proto-value functions are task-independent value
functions: they capture ``bottlenecks'', symmetries, and other types
of geometric invariants; they also form an orthonormal set of basis
functions for low-dimensional approximation of task-specific value
functions. I also describe a novel variant of Howard's classic policy
iteration method called Representation Policy Iteration: RPI
interleaves representation and policy learning. RPI outperforms
Koller-Parr-Lagoudakis' least-squares policy iteration used with
handcoded Legendre polynomials and radial basis functions. RPI's
strength is that it can discover basis functions that exploit the
geometry of an environment instead of relying on a handcoded function
approximator.
Above all, this talk will illustrate the growing importance of
coordinate-free mathematics in AI and CS. In addition to benchmark AI
and CS problems, I will use historical anecdotes to "liven" the talk:
from Greek mythology and the agony of unrequited love, to designing a
wine cellar, finding patterns in music, and harmonic analysis of the
U.Mass CS faculty collaboration graph.