Timezone: »

 
Spotlight
Effective Diversity in Population Based Reinforcement Learning
Jack Parker-Holder · Aldo Pacchiano · Krzysztof M Choromanski · Stephen J Roberts

Thu Dec 10 07:20 AM -- 07:30 AM (PST) @ Orals & Spotlights: Reinforcement Learning
in Orals & Spotlights Track 31: Reinforcement Learning »

Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment. With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors. This behavioral diversity is often boosted via multi-objective loss functions. However, those approaches typically leverage mean field updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy. In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards. As such, the reward-diversity trade off typically relies on heuristics. Finally, such methods require behavioral representations, often handcrafted and domain specific. In this paper, we introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, defined by task-agnostic behavioral embeddings. In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques. We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.

Author Information

Jack Parker-Holder (University of Oxford)
Aldo Pacchiano (UC Berkeley)
Krzysztof M Choromanski (Google Brain Robotics)
Stephen J Roberts (University of Oxford)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors