`

Timezone: »

 
Poster
Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations
Debraj Basu · Deepesh Data · Can Karakus · Suhas Diggavi

Tue Dec 10 05:30 PM -- 07:30 PM (PST) @ East Exhibition Hall B + C #32
in Algorithms -- Large Scale Learning »

Communication bottleneck has been identified as a significant issue in distributed optimization of large-scale learning models. Recently, several approaches to mitigate this problem have been proposed, including different forms of gradient compression or computing local models and mixing them iteratively. In this paper we propose Qsparse-local-SGD algorithm, which combines aggressive sparsification with quantization and local computation along with error compensation, by keeping track of the difference between the true and compressed gradients. We propose both synchronous and asynchronous implementations of Qsparse-local-SGD. We analyze convergence for Qsparse-local-SGD in the distributed case, for smooth non-convex and convex objective functions. We demonstrate that Qsparse-local-SGD converges at the same rate as vanilla distributed SGD for many important classes of sparsifiers and quantizers. We use Qsparse-local-SGD to train ResNet-50 on ImageNet, and show that it results in significant savings over the state-of-the-art, in the number of bits transmitted to reach target accuracy.

Keywords: Algorithms · Large Scale Learning · Algorithms -> Stochastic Methods; Deep Learning -> Efficient Training Methods; Deep Learning · Optimization for Deep Networks;

Author Information

Debraj Basu (Adobe Inc.)
Deepesh Data (UCLA)
Can Karakus (Amazon Web Services)
Suhas Diggavi (UCLA)

More from the Same Authors