Timezone: »

 
Poster
Causal Influence Detection for Improving Efficiency in Reinforcement Learning
Maximilian Seitzer · Bernhard Schölkopf · Georg Martius

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @
in Poster Session 6 »

Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of situation-dependent causal influence based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.

Keywords: Reinforcement Learning and Planning · Causality

Author Information

Maximilian Seitzer (Max Planck Institute for Intelligent Systems, Max-Planck Institute)
Bernhard Schölkopf (MPI for Intelligent Systems, Tübingen)
Georg Martius (IST Austria)

More from the Same Authors