Timezone: »

 
Poster
Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
Kate Rakelly · Abhishek Gupta · Carlos Florensa · Sergey Levine

Thu Dec 09 04:30 PM -- 06:00 PM (PST) @
in Poster Session 7 »

Mutual information (MI) maximization provides an appealing formalism for learning representations of data. In the context of reinforcement learning (RL), such representations can accelerate learning by discarding irrelevant and redundant information, while retaining the information necessary for control. Much prior work on these methods has addressed the practical difficulties of estimating MI from samples of high-dimensional observations, while comparatively less is understood about which MI objectives yield representations that are sufficient for RL from a theoretical perspective. In this paper, we formalize the sufficiency of a state representation for learning and representing the optimal policy, and study several popular MI based objectives through this lens. Surprisingly, we find that two of these objectives can yield insufficient representations given mild and common assumptions on the structure of the MDP. We corroborate our theoretical results with empirical experiments on a simulated game environment with visual observations.

Keywords: Reinforcement Learning and Planning · Representation Learning

Author Information

Kate Rakelly (UC Berkeley)
Abhishek Gupta (University of California, Berkeley)
Carlos Florensa (UC Berkeley)
Sergey Levine (UC Berkeley)

More from the Same Authors