Timezone: »

The Mean-Squared Error of Double Q-Learning
Wentao Weng · Harsh Gupta · Niao He · Lei Ying · R. Srikant

Mon Dec 07 09:00 PM -- 11:00 PM (PST) @ Poster Session 0 #150

In this paper, we establish a theoretical comparison between the asymptotic mean square errors of double Q-learning and Q-learning. Our result builds upon an analysis for linear stochastic approximation based on Lyapunov equations and applies to both tabular setting or with linear function approximation, provided that the optimal policy is unique and the algorithms converge. We show that the asymptotic mean-square error of Double Q-learning is exactly equal to that of Q-learning if Double Q-learning uses twice the learning rate of Q-learning and the output of Double Q-learning is the average of its two estimators. We also present some practical implications of this theoretical observation using simulations.

Author Information

Wentao Weng (Tsinghua University)
Harsh Gupta (University of Illinois at Urbana-Champaign)
Niao He (ETH Zurich)
Lei Ying (University of Michigan)
R. Srikant (University of Illinois at Urbana-Champaign)

More from the Same Authors