Timezone: »

 
Poster
Online Meta-Critic Learning for Off-Policy Actor-Critic Methods
Wei Zhou · Yiying Li · Yongxin Yang · Huaimin Wang · Timothy Hospedales

Tue Dec 08 09:00 PM -- 11:00 PM (PST) @ Poster Session 2 #603
in Poster Session 3 »

Off-Policy Actor-Critic (OffP-AC) methods have proven successful in a variety of continuous control tasks. Normally, the critic's action-value function is updated using temporal-difference, and the critic in turn provides a loss for the actor that trains it to take actions with higher expected return. In this paper, we introduce a flexible and augmented meta-critic that observes the learning process and meta-learns an additional loss for the actor that accelerates and improves actor-critic learning. Compared to existing meta-learning algorithms, meta-critic is rapidly learned online for a single task, rather than slowly over a family of tasks. Crucially, our meta-critic is designed for off-policy based learners, which currently provide state-of-the-art reinforcement learning sample efficiency. We demonstrate that online meta-critic learning benefits to a variety of continuous control tasks when combined with contemporary OffP-AC methods DDPG, TD3 and SAC.

Author Information

Wei Zhou (National University of Defense Technology)
Yiying Li (National University of Defense Technology)
Yongxin Yang (University of Edinburgh )
Huaimin Wang (National University of Defense Technology)
Timothy Hospedales (University of Edinburgh)

More from the Same Authors