Timezone: »

 
Poster
Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards
Yijie Guo · Jongwook Choi · Marcin Moczulski · Shengyu Feng · Samy Bengio · Mohammad Norouzi · Honglak Lee

Thu Dec 10 09:00 AM -- 11:00 AM (PST) @ Poster Session 5 #1381
in Poster Session 6 »

Reinforcement learning with sparse rewards is challenging because an agent can rarely obtain non-zero rewards and hence, gradient-based optimization of parameterized policies can be incremental and slow. Recent work demonstrated that using a memory buffer of previous successful trajectories can result in more effective policies. However, existing methods may overly exploit past successful experiences, which can encourage the agent to adopt sub-optimal and myopic behaviors. In this work, instead of focusing on good experiences with limited diversity, we propose to learn a trajectory-conditioned policy to follow and expand diverse past trajectories from a memory buffer. Our method allows the agent to reach diverse regions in the state space and improve upon the past trajectories to reach new states. We empirically show that our approach significantly outperforms count-based exploration methods (parametric approach) and self-imitation learning (parametric approach with non-parametric memory) on various complex tasks with local optima. In particular, without using expert demonstrations or resetting to arbitrary states, we achieve the state-of-the-art scores under five billion number of frames, on challenging Atari games such as Montezuma’s Revenge and Pitfall.

Author Information

Yijie Guo (University of Michigan)
Jongwook Choi (University of Michigan)
Marcin Moczulski (Google Brain)
Shengyu Feng (University of Illinois Urbana Champaign)
Samy Bengio (Apple)
Mohammad Norouzi (Google Brain)
Honglak Lee (U. Michigan)

More from the Same Authors