Timezone: »

 
Oral
Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning
Nino Vieillard · Tadashi Kozuno · Bruno Scherrer · Olivier Pietquin · Remi Munos · Matthieu Geist

Thu Dec 10 06:30 AM -- 06:45 AM (PST) @ Orals & Spotlights: Reinforcement Learning
in Orals & Spotlights Track 31: Reinforcement Learning »

Recent Reinforcement Learning (RL) algorithms making use of Kullback-Leibler (KL) regularization as a core component have shown outstanding performance. Yet, only little is understood theoretically about why KL regularization helps, so far. We study KL regularization within an approximate value iteration scheme and show that it implicitly averages q-values. Leveraging this insight, we provide a very strong performance bound, the very first to combine two desirable aspects: a linear dependency to the horizon (instead of quadratic) and an error propagation term involving an averaging effect of the estimation errors (instead of an accumulation effect). We also study the more general case of an additional entropy regularizer. The resulting abstract scheme encompasses many existing RL algorithms. Some of our assumptions do not hold with neural networks, so we complement this theoretical analysis with an extensive empirical study.

Author Information

Nino Vieillard (Google Brain)
Tadashi Kozuno (Okinawa Institute of Science and Technology)

Tadashi Kozuno is a postdoc at the University of Alberta. He obtained bachelor and master degrees on neuroscience from Osaka university, and a PhD degree from Okinawa Inst. of Sci. and Tech. His main interest lies in efficient decision making from both theoretical and biological sides.

Bruno Scherrer (INRIA)
Olivier Pietquin (Google Research Brain Team)
Remi Munos (DeepMind)
Matthieu Geist (Google Research, Brain Team)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors