Timezone: »

 
In-context Reinforcement Learning with Algorithm Distillation
Michael Laskin · Luyu Wang · Junhyuk Oh · Emilio Parisotto · Stephen Spencer · Richie Steigerwald · DJ Strouse · Steven Hansen · Angelos Filos · Ethan Brooks · Maxime Gazeau · Himanshu Sahni · Satinder Singh · Volodymyr Mnih

@
in Deep Reinforcement Learning Workshop »
Event URL: https://openreview.net/forum?id=9jsWJfk3xR »

We propose Algorithm Distillation (AD), a method for distilling reinforcement learning (RL) algorithms into neural networks by modeling their training histories with a causal sequence model. Algorithm Distillation treats learning to reinforcement learn as an across-episode sequential prediction problem. A dataset of learning histories is generated by a source RL algorithm, and then a causal transformer is trained by autoregressively predicting actions given their preceding learning histories as context. Unlike sequential policy prediction architectures that distill post-learning or expert sequences, AD is able to improve its policy entirely in-context without updating its network parameters. We demonstrate that AD can reinforcement learn in-context in a variety of environments with sparse rewards, combinatorial task structure, and pixel-based observations, and find that AD learns a more data-efficient RL algorithm than the one that generated the source data.

[ Poster]  [ OpenReview]  [ Topia] 

Author Information

Michael Laskin (DeepMind)
Luyu Wang (DeepMind)
Junhyuk Oh (DeepMind)
Emilio Parisotto (School of Computer Science, Carnegie Mellon University)
Stephen Spencer (DeepMind)
Richie Steigerwald (DeepMind)
DJ Strouse (DeepMind)
Steven Hansen (DeepMind)
Angelos Filos (DeepMind)
Ethan Brooks (University of Michigan)
Maxime Gazeau (DeepMind)
Himanshu Sahni (Georgia Institute of Technology)
Satinder Singh (DeepMind)
Volodymyr Mnih (DeepMind)

More from the Same Authors