Timezone: »

A Unified Switching System Perspective and Convergence Analysis of Q-Learning Algorithms
Donghwan Lee · Niao He

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #362

This paper develops a novel and unified framework to analyze the convergence of a large family of Q-learning algorithms from the switching system perspective. We show that the nonlinear ODE models associated with Q-learning and many of its variants can be naturally formulated as affine switching systems. Building on their asymptotic stability, we obtain a number of interesting results: (i) we provide a simple ODE analysis for the convergence of asynchronous Q-learning under relatively weak assumptions; (ii) we establish the first convergence analysis of the averaging Q-learning algorithm; and (iii) we derive a new sufficient condition for the convergence of Q-learning with linear function approximation.

Author Information

Donghwan Lee (KAIST)
Niao He (ETH Zurich)

More from the Same Authors