Timezone: »

 
Variance Double-Down: The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning
Johan Obando Ceron · Marc Bellemare · Pablo Samuel Castro

@
in Deep Reinforcement Learning Workshop »
Event URL: https://openreview.net/forum?id=svB4_0dObPj »

In deep reinforcement learning, multi-step learning is almost unavoidable to achieve state-of-the-art performance. However, the increased variance that multistep learning brings makes it difficult to increase the update horizon beyond relatively small numbers. In this paper, we report the counterintuitive finding that decreasing the batch size parameter improves the performance of many standard deep RL agents that use multi-step learning. It is well-known that gradient variance decreases with increasing batch sizes, so obtaining improved performance by increasing variance on two fronts is a rather surprising finding. We conduct a broad set of experiments to better understand what we call the variance doubledown phenomenon.

[ Poster]  [ OpenReview]  [ Topia] 

Author Information

Johan Obando Ceron (Mila / Université de Montréal)
Marc Bellemare (Google Brain)
Pablo Samuel Castro (Google)

More from the Same Authors