Timezone: »

 
Poster
Semantic Exploration from Language Abstractions and Pretrained Representations
Allison Tam · Neil Rabinowitz · Andrew Lampinen · Nicholas Roy · Stephanie Chan · DJ Strouse · Jane Wang · Andrea Banino · Felix Hill

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #409
in Poster Session 3 »

Effective exploration is a challenge in reinforcement learning (RL). Novelty-based exploration methods can suffer in high-dimensional state spaces, such as continuous partially-observable 3D environments. We address this challenge by defining novelty using semantically meaningful state abstractions, which can be found in learned representations shaped by natural language. In particular, we evaluate vision-language representations, pretrained on natural image captioning datasets. We show that these pretrained representations drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by considering the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach with on- and off-policy RL algorithms and in two very different task domains---one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments.

Author Information

Allison Tam (DeepMind)
Neil Rabinowitz (DeepMind)
Andrew Lampinen (DeepMind)
Nicholas Roy (DeepMind)
Stephanie Chan (DeepMind)
DJ Strouse (DeepMind)
Jane Wang (DeepMind)

Jane Wang is a research scientist at DeepMind on the neuroscience team, working on meta-reinforcement learning and neuroscience-inspired artificial agents. Her background is in physics, complex systems, and computational and cognitive neuroscience.

Andrea Banino (DeepMind)
Felix Hill (Deepmind)

More from the Same Authors