Timezone: »

 
Poster
The Phenomenon of Policy Churn
Tom Schaul · Andre Barreto · John Quan · Georg Ostrovski

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #907
in Poster Session 1 »
We identify and study the phenomenon of policy churn, that is, the rapid change of the greedy policy in value-based reinforcement learning. Policy churn operates at a surprisingly rapid pace, changing the greedy action in a large fraction of states within a handful of learning updates (in a typical deep RL set-up such as DQN on Atari). We characterise the phenomenon empirically, verifying that it is not limited to specific algorithm or environment properties. A number of ablations help whittle down the plausible explanations on why churn occurs to just a handful, all related to deep learning. Finally, we hypothesise that policy churn is a beneficial but overlooked form of implicit exploration that casts $\epsilon$-greedy exploration in a fresh light, namely that $\epsilon$-noise plays a much smaller role than expected.
Keywords: Deep Learning · Reinforcement Learning · Exploration · Stability · deep rl · Policy space
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Tom Schaul (DeepMind)
Andre Barreto (DeepMind)
John Quan (Google DeepMind)
Georg Ostrovski (DeepMind)

More from the Same Authors