Andrew G. Barto

Professor Emeritus
Retired Co-Director Autonomous Learning Laboratory

College of Information and Computer Sciences
University of Massachusetts Amherst
barto [at] cs [dot] umass [dot] edu

Ph.D. students

• Chris Vigorito, 2016
• Philip Thomas, 2015
• Bruno Castro da Silva, 2015
• Will Dabney, 2014
• Scott Niekum, 2013
• Yariv Levy, 2012 (co-chair with J. Meyer)
• Scott Kuindersma, 2012 (co-chair with Rod Grupen)
• George Konidaris, 2010
• Alicia "Pippin" Peregrin Wolfe, 2010
• Özgür Şimşek, 2008
• Ashvin Shah, 2008
• Anders Jonsson, 2005
• Thomas Kalt, 2005
• Balaraman Ravindran, 2004
• Mike Rosenstein, 2003
• Ted Perkins, 2002
• Amy McGovern, 2002
• Mike Duff, 2001
• Bob Crites, 1996
• Steve Bradtke, 1994
• Satinder Singh, 1993
• Vijay Gullapalli, 1992
• Jonathan Bachrach, 1992
• Robbie Jacobs, 1990
• Stephen Judd, 1988
• Chuck Anderson, 1986
• Rich Sutton, 1984

Research Interests

My research centers on learning in machines and animals. I worked on developing learning algorithms that are useful for engineering applications but that also make contact with learning as studied by psychologists and neuroscientists. Although I make no claims to being either a psychologist or a neuroscientist, I have spent a lot of time interacting with scientists in those fields and reading their books and papers. I think it is important that new developments should be integrated as closely as possible with the state-of-the art in as many of the relevant fields as possible. It is also important to understand how new developments relate to what others have done in the past. We all occasionally “reinvent the wheel,” but we should eventually recognize that we have!

In the case of reinforcement learning (RL)—whose main ideas go back a very long way—it has been immensely gratifying to participate in establishing new links between RL and methods from the theory of stochastic optimal control. Especially exciting are the connections between temporal difference (TD) algorithms and the brain's dopamine system. These are partly responsible for rekindling my interest in RL as an approach to building and understanding autonomous agents, rather than as a collection of methods for finding good solutions to engineering problems. The second edition of the RL book with Rich Sutton contains new chapters on RL from the perspectives of psychology and neuroscience.

An area of recent interest is about what psychologists call intrinsically motivated behavior, meaning behavior that is done for its own sake rather than as a step toward solving a specific problem of clear practical value. What we learn during intrinsically motivated behavior is essential for our development as competent autonomous agents able to efficiently solve a wide range of practical problems as they arise. The idea here is that the reward signals that an RL agent learns from do not always have to come from the agent's external environment. Reward signals can also come from within the agent itself to reward behavior that helps the agent acquire knowledge and skills that will be useful over its lifetime. This leads to fundamental questions about reward signals, both extrinsic and intrinsic. What makes a good reward signal? What kinds of intrinsic reward signals do our brains, and the brains of other animals, generate? How are these signals related to evolutionary fitness, and how have they evolved? Some of my recent papers with colleagues deal with these questions.

I am also interested in the many challenges that have to be faced as RL moves out into the real world. Most important is the challenge of making sure that RL makes positive contributions to our lives that outweigh any negative consequences. Visit the Autonomous Learning Laboratory page for some more details.