Timezone: »

Workshop
The Future of Interactive Machine Learning
Kory Mathewson @korymath · Kaushik Subramanian · Mark Ho · Robert Loftin · Joseph L Austerweil · Anna Harutyunyan · Doina Precup · Layla El Asri · Matthew Gombolay · Jerry Zhu · Sonia Chernova · Charles Isbell · Patrick M Pilarski · Weng-Keen Wong · Manuela Veloso · Julie A Shah · Matthew Taylor · Brenna Argall · Michael Littman

Thu Dec 08 11:00 PM -- 09:30 AM (PST) @ Hilton Diag. Mar, Blrm. A

Interactive machine learning (IML) explores how intelligent agents solve a task together, often focusing on adaptable collaboration over the course of sequential decision making tasks. Past research in the field of IML has investigated how autonomous agents can learn to solve problems more effectively by making use of interactions with humans. Designing and engineering fully autonomous agents is a difficult and sometimes intractable challenge. As such, there is a compelling need for IML algorithms that enable artificial and human agents to collaborate and solve independent or shared goals. The range of real-world examples of IML spans from web applications such as search engines, recommendation systems and social media personalization, to dialog systems and embodied systems such as industrial robots and household robotic assistants, and to medical robotics (e.g. bionic limbs, assistive devices, and exoskeletons). As intelligent systems become more common in industry and in everyday life, the need for these systems to interact with and learn from the people around them will also increase.

This workshop seeks to brings together experts in the fields of IML, reinforcement learning (RL), human-computer interaction (HCI), robotics, cognitive psychology and the social sciences to share recent advances and explore the future of IML. Some questions of particular interest for this workshop include: How can recent advancements in machine learning allow interactive learning to be deployed in current real world applications? How do we address the challenging problem of seamless communication between autonomous agents and humans? How can we improve the ability to collaborate safely and successfully across a diverse set of users?

We hope that this workshop will produce several outcomes:
- A review of current algorithms and techniques for IML, and a focused perspective on what is lacking;
- A formalization of the main challenges for deploying modern interactive learning algorithms in the real world; and
- A forum for interdisciplinary researchers to discuss open problems and challenges, present new ideas on IML, and plan for future collaborations.

Topics relevant to this workshop include:
Human-robot interaction
Collaborative and/or shared control
Semi-supervised learning with human intervention
Learning from demonstration, interaction and/or observation
Reinforcement learning with human-in-the-loop
Active learning, Preference learning
Transfer learning (human-to-machine, machine-to-machine)
Natural language processing for dialog systems
Computer vision for human interaction with autonomous systems
Transparency and feedback in machine learning
Computational models of human teaching
Intelligent personal assistants and dialog systems
Brain-computer interfaces (e.g. human-semi-autonomous system interfaces)
Intelligent medical robots (e.g. smart wheelchairs, prosthetics, exoskeletons)

 Thu 11:20 p.m. - 12:10 a.m. Opening Remarks, Invited Talk: Michael C. Mozer (Invited Talk) Michael Mozer 🔗 Fri 12:10 a.m. - 12:30 a.m. A Human-in-the-loop Approach for Troubleshooting Machine Learning Systems, Besmira Nushi, Ece Kamar, Donald Kossmann and Eric Horvitz (Paper Presentation) We study the problem of troubleshooting machine learning systems that rely on analytical pipelines of distinct components. Understanding and fixing errors that arise in such integrative systems is difficult as failures can occur at multiple points in the execution workflow. Moreover, errors can propagate, become amplified or be suppressed, making blame assignment difficult. We propose a human-in-the-loop methodology which leverages human intellect for troubleshooting system failures. The approach simulates potential component fixes through human computation tasks and measures the expected improvements in the holistic behavior of the system. The method provides guidance to designers about how they can best improve the system. We demonstrate the effectiveness of the approach on an automated image captioning system that has been pressed into real-world use. 🔗 Fri 12:30 a.m. - 12:50 a.m. Efficient Exploration in Monte Carlo Tree Search using Human Action Abstractions, Kaushik Subramanian, Jonathan Scholz, Charles Isbell and Andrea Thomaz (Paper Presentation) Monte Carlo Tree Search (MCTS) is a family of methods for planning in large domains. It focuses on finding a good action for a particular state, making its complexity independent of the size of the state space. However such methods are exponential with respect to the branching factor. Effective application of MCTS requires good heuristics to arbitrate action selection during learning. In this paper we present a policy-guided approach that utilizes action abstractions, derived from human input, with MCTS to facilitate efficient exploration. We draw from existing work in hierarchical reinforcement learning, interactive machine learning and show how multi-step actions, represented as stochastic policies, can serve as good action selection heuristics. We demonstrate the efficacy of our approach in the PacMan domain and highlight its advantages over traditional MCTS. 🔗 Fri 12:50 a.m. - 1:30 a.m. Invited Talk: Mattew E. Taylor (Invited Talk) Matthew Taylor 🔗 Fri 1:30 a.m. - 2:00 a.m. Coffee Break 1 (Break) 🔗 Fri 2:00 a.m. - 2:40 a.m. Invited Talk: Olivier Pietquin (Invited Talk) Olivier Pietquin 🔗 Fri 2:40 a.m. - 3:10 a.m. Poster Spotlight Talks 1 (Poster Spotlight 1) SPARC: an efficient way to combine reinforcement learning and supervised autonomy, Emmanuel Senft, Paul Baxter, Séverin Lemaignan and Tony Belpaeme Near-optimal Bayesian Active Learning with Correlated and Noisy Tests, Yuxin Chen, Hamed Hassani and Andreas Krause A Multimodal Human-Robot Interaction Dataset, Pablo Azagra, Yoan Mollard, Florian Golemo, Ana Cristina Murillo, Manuel Lopes and Javier Civera Cross-Entropy as a Criterion for Robust Interactive Learning of Latent Properties, Johannes Kulick, Robert Lieck and Marc Toussaint Ensemble Co-Training of Image and EEG-based RSVP Classifiers for Improved Image Triage, Steven Gutstein, Vernon Lawhern and Brent Lance Active Reinforcement Learning: Observing Rewards at a Cost, David Krueger, Owain Evans, Jan Leike and John Salvatier ReVACNN: Steering Convolutional Neural Network via Real-Time Visual Analytics, Sunghyo Chung, Cheonbok Park, Sangho Suh, Kyeongpil Kang, Jaegul Choo and Bum Chul Kwon Analysis of a Design Pattern for Teaching with Features and Labels, Christopher Meek, Patrice Simard and Jerry Zhu Agent-Agnostic Human-in-the-Loop Reinforcement Learning, David Abel, Owain Evans, John Salvatier and Andreas Stuhlmüller 🔗 Fri 3:10 a.m. - 3:50 a.m. Invited Talk: Todd Gureckis (Invited Talk) Todd Gureckis 🔗 Fri 3:50 a.m. - 5:00 a.m. Lunch Break (Break) 🔗 Fri 5:00 a.m. - 5:30 a.m. Poster Spotlight Talks 2 (Poster Spotlight 2) Probabilistic Expert Knowledge Elicitation of Feature Relevances in Sparse Linear Regression, Pedram Daee, Tomi Peltola, Marta Soare and Samuel Kaski Socratic Learning, Paroma Varma, Rose Yu, Dan Iter, Chris De Sa and Christopher Re Probabilistic Active Learning for Active Class Selection, Daniel Kottke, Georg Krempl, Marianne Stecklina, Cornelius Styp von Rekowski, Tim Sabsch, Tuan Pham Minh, Matthias Deliano, Myra Spiliopoulou and Bernhard Sick Regression Analysis in Small-n-Large-p Using Interactive Prior Elicitation of Pairwise Similarities, Homayun Afrabandpey, Tomi Peltola and Samuel Kaski Scalable batch mode Optimal Experimental Design for Deep Networks, Mélanie Ducoffe, Geoffrey Portelli and Frederic Precioso Interactive Preference Learning of Utility Functions for Multi-Objective Optimization, Ian Dewancker, Michael Mccourt and Samuel Ainsworth Improving Online Learning of Visual Categories by Deep Features, Lydia Fischer, Stephan Hasler, Sebastian Schrom and Heiko Wersing Interactive user intent modeling for eliciting priors of a normal linear model, Iiris Sundin, Luana Micallef, Pekka Marttinen, Muhammad Ammad-Ud-Din, Tomi Peltola, Marta Soare, Giulio Jacucci and Samuel Kaski Training an Interactive Humanoid Robot Using Multimodal Deep Reinforcement Learning, Heriberto Cuayahuitl, Guillaume Couly and Clement Olalainty 🔗 Fri 5:30 a.m. - 6:10 a.m. Invited Talk: Aude Billard (Invited Talk) Aude G Billard 🔗 Fri 6:10 a.m. - 6:30 a.m. Coffee Break 2 (Break) 🔗 Fri 6:30 a.m. - 7:30 a.m. Poster Session 🔗 Fri 7:30 a.m. - 7:50 a.m. Enabling Robots to Communicate Reward Functions, Sandy Huang, David Held, Pieter Abbeel and Anca Dragan (Paper Presentation) Understanding a robot's reward function is key to anticipating how the robot will act in a new situation. Our goal is to generate a set of robot behaviors that best illustrates a robot's reward function. We build on prior work modeling inference of the reward function from example behavior via Inverse Reinforcement Learning (IRL). Prior work using IRL has focused on people teaching machines and assumes exact inference. Our insight is that when teaching people, they will not perform exact inference. We show that while leveraging models of noisy inference can be beneficial, it is also important to achieve coverage in the space of possible strategies the robot can use. We introduce a hybrid algorithm that targets informative examples via both a noisy inference model and coverage. 🔗 Fri 7:50 a.m. - 8:10 a.m. Hierarchical Multi-Agent Reinforcement Learning through Communicative Actions for Human-Robot Collaboration, Elena Corina Grigore and Brian Scassellati (Paper Presentation) As we expect robots to start moving from working in isolated industry settings into human populated environments, our need to develop suitable learning algorithms for the latter increases. Human-robot collaboration is a particular area that has tremendous gains from endowing a robot with such learning capabilities, focusing on robots that can work side-by-side with a human and provide supportive behaviors throughout a task executed by the human worker. In this paper, we propose a framework based on hierarchical multi-agent reinforcement learning that considers the human as an expert'' agent in the system—an agent whose actions we cannot control but whose actions, jointly with the robot's actions, impact the state of the task. Our framework aims to provide the learner (the robot) with a way of learning how to provide supportive behaviors to the expert agent (the person) during a complex task. The robot employs communicative actions to interactively learn from the expert agent at key points during the task. We use a hierarchical approach in order to integrate the communicative actions in the multi-agent reinforcement learning framework and allow for simultaneously learning the quality of performing different supportive behaviors for particular combinations of task states and expert agent actions. In this paper, we present our proposed framework, detail the motion capture system data collection we performed in order to learn about the task states and characterize the expert agent's actions, and discuss how we can apply the framework to our human-robot collaboration scenario. 🔗 Fri 8:10 a.m. - 8:50 a.m. Invited Talk: Emma Brunskill (Invited Talk) Emma Brunskill 🔗 Fri 8:50 a.m. - 9:40 a.m. Panel Discussion, Closing Remarks (Discussion Panel) 🔗

#### Author Information

##### Joseph L Austerweil (University of Wisconsin, Madison)

As a computational cognitive psychologist, my research program explores questions at the intersection of perception and higher-level cognition. I use recent advances in statistics and computer science to formulate ideal learner models to see how they solve these problems and then test the model predictions using traditional behavioral experimentation. Ideal learner models help us understand the knowledge people use to solve problems because such knowledge must be made explicit for the ideal learner model to successfully produce human behavior. This method yields novel machine learning methods and leads to the discovery of new psychological principles.

##### Charles Isbell (Georgia Tech)

Dr. Charles Isbell received his bachelor's in Information and Computer Science from Georgia Tech, and his MS and PhD at MIT's AI Lab. Upon graduation, he worked at AT&T Labs/Research until 2002, when he returned to Georgia Tech to join the faculty as an Assistant Professor. He has served many roles since returning and is now The John P. Imlay Jr. Dean of the College of Computing. Charles’s research interests are varied but the unifying theme of his work has been using machine learning to build autonomous agents who engage directly with humans. His work has been featured in the popular press, congressional testimony, and in several technical collections. In parallel, Charles has also pursued reform in computing education. He was a chief architect of Threads, Georgia Tech’s structuring principle for computing curricula. Charles was also an architect for Georgia Tech’s First-of-its’s-kind MOOC-supported MS in Computer Science. Both efforts have received international attention, and been presented in the academic and popular press. In all his roles, he has continued to focus on issues of broadening participation in computing, and is the founding Executive Director for the Constellations Center for Equity in Computing. He is an AAAI Fellow and a Fellow of the ACM. Appropriately, his citation for ACM Fellow reads “for contributions to interactive machine learning; and for contributions to increasing access and diversity in computing”.