Timezone: »

 
Poster
Unsupervised Curricula for Visual Meta-Reinforcement Learning
Allan Jabri · Kyle Hsu · Abhishek Gupta · Benjamin Eysenbach · Sergey Levine · Chelsea Finn

Wed Dec 11 05:00 PM -- 07:00 PM (PST) @ East Exhibition Hall B + C #35
in Algorithms -- Meta-Learning »

In principle, meta-reinforcement learning algorithms leverage experience across many tasks to learn fast reinforcement learning (RL) strategies that transfer to similar tasks. However, current meta-RL approaches rely on manually-defined distributions of training tasks, and hand-crafting these task distributions can be challenging and time-consuming. Can ``useful'' pre-training tasks be discovered in an unsupervised manner? We develop an unsupervised algorithm for inducing an adaptive meta-training task distribution, i.e. an automatic curriculum, by modeling unsupervised interaction in a visual environment. The task distribution is scaffolded by a parametric density model of the meta-learner's trajectory distribution. We formulate unsupervised meta-RL as information maximization between a latent task variable and the meta-learner’s data distribution, and describe a practical instantiation which alternates between integration of recent experience into the task distribution and meta-learning of the updated tasks. Repeating this procedure leads to iterative reorganization such that the curriculum adapts as the meta-learner's data distribution shifts. In particular, we show how discriminative clustering for visual representation can support trajectory-level task acquisition and exploration in domains with pixel observations, avoiding pitfalls of alternatives. In experiments on vision-based navigation and manipulation domains, we show that the algorithm allows for unsupervised meta-learning that transfers to downstream tasks specified by hand-crafted reward functions and serves as pre-training for more efficient supervised meta-learning of test task distributions.

Keywords: Unsupervised Learning · Reinforcement Learning · Reinforcement Learning and Planning · Algorithms -> Meta-Learning; Algorithms
[ Paper]  [ Poster]  [ Slides] 

Author Information

Allan Jabri (UC Berkeley)
Kyle Hsu (University of Toronto)
Abhishek Gupta (University of California, Berkeley)
Benjamin Eysenbach (Carnegie Mellon University)
Benjamin Eysenbach

I'm a 5th year PhD student at CMU, focusing on RL algorithms. I am currently on the faculty job market.

Sergey Levine (UC Berkeley)
Chelsea Finn (Stanford University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors