Timezone: »

 
Poster
Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
Shuai Zheng · Ziyue Huang · James Kwok

Thu Dec 12 05:00 PM -- 07:00 PM (PST) @ East Exhibition Hall B + C #211
in Optimization -- Non-Convex Optimization »
Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote achieves a 32x reduction in communication cost. However, its convergence is based on unrealistic assumptions and can diverge in practice. In this paper, we propose a general distributed compressed SGD with Nesterov's momentum. We consider two-way compression, which compresses the gradients both to and from workers. Convergence analysis on nonconvex problems for general gradient compressors is provided. By partitioning the gradient into blocks, a blockwise compressor is introduced such that each gradient block is compressed and transmitted in 1-bit format with a scaling factor, leading to a nearly 32x reduction on communication. Experimental results show that the proposed method converges as fast as full-precision distributed momentum SGD and achieves the same testing accuracy. In particular, on distributed ResNet training with 7 workers on the ImageNet, the proposed algorithm achieves the same testing accuracy as momentum SGD using full-precision gradients, but with $46\%$ less wall clock time.
Keywords: Distributed Inference · Probabilistic Methods · Algorithms -> Large Scale Learning; Deep Learning -> Efficient Training Methods; Deep Learning · Optimization for Deep Network
[ Paper]  [ Poster] 

Author Information

Shuai Zheng (Hong Kong University of Science and Technology / Amazon Web Services)
Ziyue Huang (Hong Kong University of Science and Technology)
James Kwok (Hong Kong University of Science and Technology)

More from the Same Authors