Skip to yearly menu bar Skip to main content

Workshop: Deep Reinforcement Learning Workshop

Variance Double-Down: The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning

Johan Obando Ceron · Marc Bellemare · Pablo Samuel Castro


In deep reinforcement learning, multi-step learning is almost unavoidable to achieve state-of-the-art performance. However, the increased variance that multistep learning brings makes it difficult to increase the update horizon beyond relatively small numbers. In this paper, we report the counterintuitive finding that decreasing the batch size parameter improves the performance of many standard deep RL agents that use multi-step learning. It is well-known that gradient variance decreases with increasing batch sizes, so obtaining improved performance by increasing variance on two fronts is a rather surprising finding. We conduct a broad set of experiments to better understand what we call the variance doubledown phenomenon.

Chat is not available.