Skip to yearly menu bar Skip to main content


Poster

Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updates

Guangchen Lan · Han Wang · James Anderson · Christopher Brinton · Vaneet Aggarwal

Great Hall & Hall B1+B2 (level 1) #1403

Abstract: Federated reinforcement learning (FedRL) enables agents to collaboratively train a global policy without sharing their individual data. However, high communication overhead remains a critical bottleneck, particularly for natural policy gradient (NPG) methods, which are second-order. To address this issue, we propose the FedNPG-ADMM framework, which leverages the alternating direction method of multipliers (ADMM) to approximate global NPG directions efficiently. We theoretically demonstrate that using ADMM-based gradient updates reduces communication complexity from O(d2) to O(d) at each iteration, where d is the number of model parameters. Furthermore, we show that achieving an ϵ-error stationary convergence requires O(1(1γ)2ϵ) iterations for discount factor γ, demonstrating that FedNPG-ADMM maintains the same convergence rate as standard FedNPG. Through evaluation of the proposed algorithms in MuJoCo environments, we demonstrate that FedNPG-ADMM maintains the reward performance of standard FedNPG, and that its convergence rate improves when the number of federated agents increases.

Chat is not available.