Timezone: »
Poster
Distributed Principal Component Analysis with Limited Communication
Foivos Alimisis · Peter Davies · Bart Vandereycken · Dan Alistarh
We study efficient distributed algorithms for the fundamental problem of principal component analysis and leading eigenvector computation on the sphere, when the data are randomly distributed among a set of computational nodes. We propose a new quantized variant of Riemannian gradient descent to solve this problem, and prove that the algorithm converges with high probability under a set of necessary spherical-convexity properties. We give bounds on the number of bits transmitted by the algorithm under common initialization schemes, and investigate the dependency on the problem dimension in each case.
Author Information
Foivos Alimisis (IST Austria)
Peter Davies (University of Surrey)
Bart Vandereycken (Princeton University)
Dan Alistarh (IST Austria & NeuralMagic)
More from the Same Authors
-
2021 : SSSE: Efficiently Erasing Samples from Trained Machine Learning Models »
Alexandra Peste · Dan Alistarh · Christoph Lampert -
2022 Poster: Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning »
Elias Frantar · Dan Alistarh -
2021 Poster: M-FAC: Efficient Matrix-Free Approximations of Second-Order Information »
Elias Frantar · Eldar Kurtic · Dan Alistarh -
2021 Poster: Towards Tight Communication Lower Bounds for Distributed Optimisation »
Janne H. Korhonen · Dan Alistarh -
2021 Poster: Asynchronous Decentralized SGD with Quantized and Local Updates »
Giorgi Nadiradze · Amirmojtaba Sabour · Peter Davies · Shigang Li · Dan Alistarh -
2021 Poster: AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks »
Alexandra Peste · Eugenia Iofinova · Adrian Vladu · Dan Alistarh -
2019 Poster: Powerset Convolutional Neural Networks »
Chris Wendler · Markus Püschel · Dan Alistarh -
2017 Poster: QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding »
Dan Alistarh · Demjan Grubic · Jerry Li · Ryota Tomioka · Milan Vojnovic -
2017 Spotlight: Communication-Efficient Stochastic Gradient Descent, with Applications to Neural Networks »
Dan Alistarh · Demjan Grubic · Jerry Li · Ryota Tomioka · Milan Vojnovic -
2014 Workshop: Riemannian geometry in machine learning, statistics and computer vision »
Minh Ha Quang · Vikas Sindhwani · Vittorio Murino · Michael Betancourt · Tom Fletcher · Richard I Hartley · Anuj Srivastava · Bart Vandereycken