Machine Learning and Friends Lunch

home
past talks
resources
conferences

A novel approach to abstraction discovery in MDPs

Abstract

As the size of control problems grow, it becomes increasingly difficult for learning methods to successfully learn an optimal or near-optimal behavior. In many cases it is useful to perform abstraction, reducing the size of the parameter space in order to accelerate learning. Performing abstraction requires detailed knowledge about a control problem that may not be available prior to learning. If abstractions are discovered from experience, it is possible to accelerate learning in large control problems even if prior knowledge is sparse.

I will present a novel approach to abstraction discovery in Markov decision processes that takes advantage of conditional independence between the variables describing a control problem. I assume that an MDP is factored and use a dynamic Bayes network representation to estimate transition probabilities and expected reward. Since I do not want to assume that detailed prior knowledge is available, I have to learn the DBN representation from experience. I will demonstrate how the DBN representation enables a principled approach to performing abstraction in MDPs.