Timezone: »
Understanding emerging behaviors of reinforcement learning (RL) agents may be difficult since such agents are often trained in complex environments using highly complex decision making procedures. This has given rise to a variety of approaches to explainability in RL that aim to reconcile discrepancies that may arise between the behavior of an agent and the behavior that is anticipated by an observer. Most recent approaches have relied either on domain knowledge, that may not always be available, on an analysis of the agent’s policy, or on an analysis of specific elements of the underlying environment, typically modeled as a Markov Decision Process (MDP). Our key claim is that even if the underlying model is not fully known (e.g., the transition probabilities have not been accurately learned) or is not maintained by the agent (i.e., when using model-free methods), the model can nevertheless be exploited to automatically generate explanations. For this purpose, we suggest using formal MDP abstractions and transforms, previously used in the literature for expediting the search for optimal policies, to automatically produce explanations. Since such transforms are typically based on a symbolic representation of the environment, they can provide meaningful explanations for gaps between the anticipated and actual agent behavior. We formally define the explainability problem, suggest a class of transforms that can be used for explaining emergent behaviors, and suggest methods that enable efficient search for an explanation. We demonstrate the approach on a set of standard benchmarks.
Author Information
Mira Finkelstein (The Hebrew University)
Nitsan levy (Hebrew University of Jerusalem)
Lucy Liu (Harvard University)
Yoav Kolumbus (Hebrew University of Jerusalem)
David Parkes (Harvard University)
David C. Parkes is Gordon McKay Professor of Computer Science in the School of Engineering and Applied Sciences at Harvard University. He was the recipient of the NSF Career Award, the Alfred P. Sloan Fellowship, the Thouron Scholarship and the Harvard University Roslyn Abramson Award for Teaching. Parkes received his Ph.D. degree in Computer and Information Science from the University of Pennsylvania in 2001, and an M.Eng. (First class) in Engineering and Computing Science from Oxford University in 1995. At Harvard, Parkes leads the EconCS group and teaches classes in artificial intelligence, optimization, and topics at the intersection between computer science and economics. Parkes has served as Program Chair of ACM EC’07 and AAMAS’08 and General Chair of ACM EC’10, served on the editorial board of Journal of Artificial Intelligence Research, and currently serves as Editor of Games and Economic Behavior and on the boards of Journal of Autonomous Agents and Multi-agent Systems and INFORMS Journal of Computing. His research interests include computational mechanism design, electronic commerce, stochastic optimization, preference elicitation, market design, bounded rationality, computational social choice, networks and incentives, multi-agent systems, crowd-sourcing and social computing.
Jeffrey S Rosenschein (The Hebrew University of Jerusalem)
Sarah Keren (Technion, Technion)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: Explainable Reinforcement Learning via Model Transforms »
Thu. Dec 1st through Fri the 2nd Room Hall J #736
More from the Same Authors
-
2021 : Deep Reinforcement Learning Explanation via Model Transforms »
Sarah Keren · Yoav Kolumbus · Jeffrey S Rosenschein · David Parkes · Mira Finkelstein -
2021 : Promoting Resilience of Multi-Agent Reinforcement Learning via Confusion-Based Communication »
Ofir Abu · Sarah Keren · Matthias Gerstgrasser · Jeffrey S Rosenschein -
2021 : Promoting Resilience in Multi-Agent Reinforcement Learning via Confusion-Based Communication »
Ofir Abu · Matthias Gerstgrasser · Jeffrey S Rosenschein · Sarah Keren -
2021 : Promoting Resilience of Multi-Agent Reinforcement Learning via Confusion-Based Communication »
Ofir Abu · Sarah Keren · Matthias Gerstgrasser · Jeffrey S Rosenschein -
2022 Poster: How and Why to Manipulate Your Own Agent: On the Incentives of Users of Learning Agents »
Yoav Kolumbus · Noam Nisan -
2022 : Enhancing Transfer of Reinforcement Learning Agents with Abstract Contextual Embeddings »
Guy Azran · Mohamad Hosein Danesh · Stefano Albrecht · Sarah Keren -
2022 : Predictive Multiplicity in Probabilistic Classification »
Jamelle Watson-Daniels · David Parkes · Berk Ustun -
2022 : Predictive Multiplicity in Probabilistic Classification »
Jamelle Watson-Daniels · David Parkes · Berk Ustun -
2022 : Learning to Mitigate AI Collusion on E-Commerce Platforms »
Eric Mibuari · Gianluca Brero · David Parkes · Nicolas Lepore -
2022 : Meta-RL for Multi-Agent RL: Learning to Adapt to Evolving Agents »
Matthias Gerstgrasser · David Parkes -
2022 : Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning »
Matthias Gerstgrasser · Tom Danino · Sarah Keren -
2022 : A (dis-)information theory of revealed and unrevealed preferences »
Nitay Alon · Lion Schulz · Peter Dayan · Jeffrey S Rosenschein -
2023 Poster: Data Market Design through Deep Learning »
Sai Srivatsa Ravindranath · Yanchen Jiang · David Parkes -
2023 Poster: Deep Contract Design via Discontinuous Piecewise Affine Neural Networks »
Tonghan Wang · Paul Duetting · Dmitry Ivanov · Inbal Talgam-Cohen · David Parkes -
2023 Poster: Asynchronous Proportional Response Dynamics in Markets with Adversarial Scheduling »
Yoav Kolumbus · Menahem Levy · Noam Nisan -
2022 Spotlight: Lightning Talks 5A-4 »
Yangrui Chen · Zhiyang Chen · Liang Zhang · Hanqing Wang · Jiaqi Han · Shuchen Wu · shaohui peng · Ganqu Cui · Yoav Kolumbus · Noemi Elteto · Xing Hu · Anwen Hu · Wei Liang · Cong Xie · Lifan Yuan · Noam Nisan · Wenbing Huang · Yousong Zhu · Ishita Dasgupta · Luc V Gool · Tingyang Xu · Rui Zhang · Qin Jin · Zhaowen Li · Meng Ma · Bingxiang He · Yangyi Chen · Juncheng Gu · Wenguan Wang · Ke Tang · Yu Rong · Eric Schulz · Fan Yang · Wei Li · Zhiyuan Liu · Jiaming Guo · Yanghua Peng · Haibin Lin · Haixin Wang · Qi Yi · Maosong Sun · Ruizhi Chen · Chuan Wu · Chaoyang Zhao · Yibo Zhu · Liwei Wu · xishan zhang · Zidong Du · Rui Zhao · Jinqiao Wang · Ling Li · Qi Guo · Ming Tang · Yunji Chen -
2022 Spotlight: Lightning Talks 5A-2 »
Qiang LI · Zhiwei Xu · Jia-Qi Yang · Thai Hung Le · Haoxuan Qu · Yang Li · Artyom Sorokin · Peirong Zhang · Mira Finkelstein · Nitsan levy · Chung-Yiu Yau · dapeng li · Thommen Karimpanal George · De-Chuan Zhan · Nazar Buzun · Jiajia Jiang · Li Xu · Yichuan Mo · Yujun Cai · Yuliang Liu · Leonid Pugachev · Bin Zhang · Lucy Liu · Hoi-To Wai · Liangliang Shi · Majid Abdolshah · Yoav Kolumbus · Lin Geng Foo · Junchi Yan · Mikhail Burtsev · Lianwen Jin · Yuan Zhan · Dung Nguyen · David Parkes · Yunpeng Baiia · Jun Liu · Kien Do · Guoliang Fan · Jeffrey S Rosenschein · Sunil Gupta · Sarah Keren · Svetha Venkatesh -
2022 Spotlight: How and Why to Manipulate Your Own Agent: On the Incentives of Users of Learning Agents »
Yoav Kolumbus · Noam Nisan -
2022 : A (dis-)information theory of revealed and unrevealed preferences »
Nitay Alon · Lion Schulz · Peter Dayan · Jeffrey S Rosenschein -
2022 Poster: Learning to Mitigate AI Collusion on Economic Platforms »
Gianluca Brero · Eric Mibuari · Nicolas Lepore · David Parkes -
2021 Workshop: Learning in Presence of Strategic Behavior »
Omer Ben-Porat · Nika Haghtalab · Annie Liang · Yishay Mansour · David Parkes -
2020 : Panel: Kate Larson (DeepMind) [moderator], Natasha Jaques (Google), Jeffrey Rosenschein (The Hebrew University of Jerusalem), Michael Wooldridge (University of Oxford) »
Kate Larson · Natasha Jaques · Jeffrey S Rosenschein · Michael Wooldridge -
2020 Workshop: Machine Learning for Economic Policy »
Stephan Zheng · Alexander Trott · Annie Liang · Jamie Morgenstern · David Parkes · Nika Haghtalab -
2020 Poster: From Predictions to Decisions: Using Lookahead Regularization »
Nir Rosenfeld · Anna Hilgard · Sai Srivatsa Ravindranath · David Parkes -
2019 Poster: Finding Friend and Foe in Multi-Agent Games »
Jack Serrino · Max Kleiman-Weiner · David Parkes · Josh Tenenbaum -
2019 Spotlight: Finding Friend and Foe in Multi-Agent Games »
Jack Serrino · Max Kleiman-Weiner · David Parkes · Josh Tenenbaum -
2017 : Optimal Economic Design through Deep Learning »
David Parkes -
2017 Poster: Multi-View Decision Processes: The Helper-AI Problem »
Christos Dimitrakakis · David Parkes · Goran Radanovic · Paul Tylkin -
2016 Poster: Long-term Causal Effects via Behavioral Game Theory »
Panagiotis Toulis · David Parkes -
2015 Poster: Learnability of Influence in Networks »
Harikrishna Narasimhan · David Parkes · Yaron Singer -
2014 Workshop: NIPS’14 Workshop on Crowdsourcing and Machine Learning »
David Parkes · Denny Zhou · Chien-Ju Ho · Nihar Bhadresh Shah · Adish Singla · Jared Heyman · Edwin Simpson · Andreas Krause · Rafael Frongillo · Jennifer Wortman Vaughan · Panagiotis Papadimitriou · Damien Peters -
2014 Workshop: Analysis of Rank Data: Confluence of Social Choice, Operations Research, and Machine Learning »
Shivani Agarwal · Hossein Azari Soufiani · Guy Bresler · Sewoong Oh · David Parkes · Arun Rajkumar · Devavrat Shah -
2014 Workshop: NIPS Workshop on Transactional Machine Learning and E-Commerce »
David Parkes · David H Wolpert · Jennifer Wortman Vaughan · Jacob D Abernethy · Amos Storkey · Mark Reid · Ping Jin · Nihar Bhadresh Shah · Mehryar Mohri · Luis E Ortiz · Robin Hanson · Aaron Roth · Satyen Kale · Sebastien Lahaie -
2014 Poster: A Statistical Decision-Theoretic Framework for Social Choice »
Hossein Azari Soufiani · David Parkes · Lirong Xia -
2014 Oral: A Statistical Decision-Theoretic Framework for Social Choice »
Hossein Azari Soufiani · David Parkes · Lirong Xia -
2013 Poster: Generalized Random Utility Models with Multiple Types »
Hossein Azari Soufiani · Hansheng Diao · Zhenyu Lai · David Parkes -
2013 Poster: Contrastive Learning Using Spectral Methods »
James Y Zou · Daniel Hsu · David Parkes · Ryan Adams -
2013 Poster: Generalized Method-of-Moments for Rank Aggregation »
Hossein Azari Soufiani · William Z Chen · David Parkes · Lirong Xia -
2010 Invited Talk: The Interplay of Machine Learning and Mechanism Design »
David Parkes