Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL

Reinforcement learning (RL) is a critical component for post-training large language models (LLMs). However, in bandwidth-constrained distributed RL, scalability is often bottlenecked by the synchronization of policy weights from trainers to inference workers, particularly over commodity networks. This motivates a systematic empirical study of bitwise weight-update sparsity at both step-level and multi-step granularities, examining its evolution across training dynamics, off-policy delay, and model scale. We find that update sparsity is consistently high, frequently exceeding 99% across practically relevant settings. We demonstrate that this arises from the interaction of bfloat16 (BF16) precision and the learning rates typically used in RL. Leveraging these observations, we propose PULSESync (Precision-gated Updates for Low-precision Sparse Exchange), a lossless weight synchronization method that transmits only the indices and values of modified parameters. In bandwidth-constrained decentralized environments, our approach achieves over 100× (14 GB → ~108 MB) communication reduction while maintaining bit-identical training dynamics and performance compared to full weight synchronization. This lowers the bandwidth required to sustain high GPU utilization from 20 Gbit/s to 0.2 Gbit/s, allowing decentralized RL training to approach centralized throughput.