- Causal Reasoning for Explainability of Deep Networks
- Evaluating Saliency Map Methods in Deep Reinforcement Learning
Saliency maps have been used to support explanations of deep reinforcement learning (RL) agent behavior over temporally extended sequences. However, their use in the community indicates that the explanations derived from saliency maps are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and assess the degree to which saliency maps represent semantics of RL environments. We evaluate three types of saliency maps using Atari games, a common benchmark for deep RL. Our results show the extent to which existing claims about Atari games can be evaluated and suggest that saliency maps are an exploratory tool not an explanatory tool. This work appeared at ICLR 2020. An earlier version was accepted as a poster at the 2019 WiML Workshop co-located with NeurIPS 2019.
Akanksha Atrey, Kaleigh Clary, David Jensen
[ ICLR 2020 paper ] [ bibtex ] [ code ] [ reviews ]
- Post-Training Variability of Deep Reinforcement Learning Models
Reproducibility in deep reinforcement learning has proven challenging due to the large number of factors influencing agent performance. We find that post-training performance (measured as game score distributions) can exhibit several characteristics that make reporting common summary statistics an unsound metric for agent performance. Our experiments demonstrate the variability of common agents used in the popular OpenAI Baselines repository. We make the case for reporting post-training agent performance as a distribution, rather than a point estimate. This work was accepted for a short talk at the 2018 NeurIPS Critiquing and Correcting Trends workshop and featured in a spotlight talk.
Kaleigh Clary, Emma Tosch, John Foley, David Jensen
[ CRACT paper ] [ bibtex ] [ code ] [ slides ]
- Counterfactual Evaluations of Deep Systems
I have been supported by a DARPA XAI grant to develop methods for explaining the decisions and behavior of deep networks. Our approach uses counterfactual reasoning to identify changes in network outputs resulting from changes in human-understandable concepts from the application domain. We have applied these methods with models trained for classification (pedestrian detection) and reinforcement learning (Atari, Starcraft II).
[ UMass press release ] [ NYT ]
- Toybox: Atari Reimplementations for Interventional Experimentation in Deep Reinforcement Learning
Atari games have become the de facto benchmark suite for deep reinforcement learning. Unfortunately, these games are still black boxes that do not permit systemtic intervention on program state. Toybox is a suite of Atari games and associated testing framework for validating behavioral requirements of agents trained on Atari games. I developed an intervention interface for performing experiments in Toybox, and helped design a set of counterfactual evaluations of saliency map explanations in our work, Evaluating Saliency Map Methods in Deep Reinforcement Learning. Toybox was presented at IBM AI Systems Day and as a poster at the 2018 NeurIPS Systems for ML Workshop.
Emma Tosch, John Foley, Kaleigh Clary, David Jensen
[ Systems4ML paper ] [ bibtex ] [ code] [ preprint ]
- A/B Testing in Networks with Adversarial Nodes
Causal estimation over large-scale relational systems requires careful experimental design, as the treatment compliance and response of an individual may be influenced by the outcomes of other inidividuals. In some cases, members of the relational network may be targeting their output to influence their neighbors. These adversarial nodes can influence effect estimates by leveraging peer effects to influence the outcomes of their neighbors, yet these nodes may not be known or detectable. We characterize the influence of adversarial nodes and the bias these nodes introduce in estimates of average treatment effect. Our work demonstrates that causal estimates in networks can be sensitive to the actions of adversaries, and we identify network structures that are particularly vulnerable to adversarial responses. This work became my Synthesis Project, which earned an Outstanding Synthesis Project award. This work was accepted for a short talk at the 2017 KDD Mining and Learning with Graphs Workshop.
Kaleigh Clary, Andrew McGregor, David Jensen
[ MLG paper ] [ bibtex ] [ code ] [ video ] [ slides ]
- Data Science for Social Good: Predicting Risk of Type II Diabetes
Data Science for Social Good is a summer fellowship program hosted at the University of Chicago (now at CMU) that brings together graduate students and young professionals representing a diverse set of skills and backgrounds to work closely with governments, nonprofits, and relevant stakeholders to develop solutions for policy and social problems across health, criminal justice, education, public safety, social services, and economic development.
Our team developed a model to identify patients at risk of developing type II diabetes. Diabetes affects over 45 million adults in the United States. It often results in additional health complications, increased health care costs, and mortality. Existing diabetes screening guidelines miss opportunities for prevention, diagnosis, and treatment among minority populations. We partnered with AllianceChicago — a national network of 44 community health centers serving the least resourced members of their communities — to identify patients at risk of developing type II diabetes so that its network of community health centers can provide better medical treatment. The de-identified data set included diagnostic codes, lab results, and geographic information for as many as two million people over the last 12 years. AllianceChicago plans to integrate the work into its electronic health records system (EHR) to help clinicians personalize their recommendations to patients and reduce their risk of developing diabetes.
Benjamin Ackerman, Kaleigh Clary, Jorge Saldivar, William Wang, Katy Dupre, Adolfo De Unánue, Elena Eneva and Rayid Ghani
[ DataFest slides ] [ code (awaiting public release) ]
|
