Timezone: »

 
Poster
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
Lin Guan · Mudit Verma · Sihang Guo · Ruohan Zhang · Subbarao Kambhampati

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @ Virtual
in Poster Session 6 »

Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form of explanation. However, this kind of human guidance was only investigated in supervised learning tasks, and it remains unclear how to best incorporate this type of human knowledge into deep reinforcement learning. In this paper, we present the first study of using human visual explanations in human-in-the-loop reinforcement learning (HIRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative "good" or "bad" feedback for queried state-action pairs, but also provides a visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through a context-aware data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, to evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning, and Human-in-the-loop RL baselines that only utilize evaluative feedback.

Keywords: Reinforcement Learning and Planning · Machine Learning

Author Information

Lin Guan (Arizona State University)
Mudit Verma (Arizona State University)
Sihang Guo (University of Texas at Austin)
Ruohan Zhang (Stanford University)
Subbarao Kambhampati (Arizona State University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors