Timezone: »
In this paper we propose a novel parallel stochastic coordinate descent (SCD) algorithm with convergence guarantees that exhibits strong scalability. We start by studying a state-of-the-art parallel implementation of SCD and identify scalability as well as system-level performance bottlenecks of the respective implementation. We then take a principled approach to develop a new SCD variant which is designed to avoid the identified system bottlenecks, such as limited scaling due to coherence traffic of model sharing across threads, and inefficient CPU cache accesses. Our proposed system-aware parallel coordinate descent algorithm (SySCD) scales to many cores and across numa nodes, and offers a consistent bottom line speedup in training time of up to x12 compared to an optimized asynchronous parallel SCD algorithm and up to x42, compared to state-of-the-art GLM solvers (scikit-learn, Vowpal Wabbit, and H2O) on a range of datasets and multi-core CPU architectures.
Author Information
Nikolas Ioannou (IBM Research)
Celestine Mendler-Dünner (UC Berkeley)
Thomas Parnell (IBM Research)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: SySCD: A System-Aware Parallel Coordinate Descent Algorithm »
Wed Dec 11th 01:30 -- 03:30 AM Room East Exhibition Hall B + C
More from the Same Authors
-
2020 Poster: Stochastic Optimization for Performative Prediction »
Celestine Mendler-Dünner · Juan Perdomo · Tijana Zrnic · Moritz Hardt -
2020 Poster: SnapBoost: A Heterogeneous Boosting Machine »
Thomas Parnell · Andreea Anghel · Małgorzata Łazuka · Nikolas Ioannou · Sebastian Kurella · Peshal Agarwal · Nikolaos Papandreou · Haralampos Pozidis -
2018 Poster: Snap ML: A Hierarchical Framework for Machine Learning »
Celestine Dünner · Thomas Parnell · Dimitrios Sarigiannis · Nikolas Ioannou · Andreea Anghel · Gummadi Ravi · Madhusudanan Kandasamy · Haralampos Pozidis -
2017 Poster: Efficient Use of Limited-Memory Accelerators for Linear Learning on Heterogeneous Systems »
Celestine Dünner · Thomas Parnell · Martin Jaggi