Timezone: »

 
Data-heterogeneity-aware Mixing for Decentralized Learning
Yatin Dandi · Anastasiia Koloskova · Martin Jaggi · Sebastian Stich

@
in OPT 2022: Optimization for Machine Learning »
Event URL: https://openreview.net/forum?id=Quz3n455QZt »

Decentralized learning provides an effective framework to train machine learning models with data distributed over arbitrary communication graphs. However, most existing approaches towards decentralized learning disregard the interaction between data heterogeneity and graph topology. In this paper, we characterize the dependence of convergence on the relationship between the mixing weights of the graph and the data heterogeneity across nodes. We propose a metric that quantifies the ability of a graph to mix the current gradients. We further prove that the metric controls the convergence rate, particularly in settings where the heterogeneity across nodes dominates the stochasticity between updates for a given node. Motivated by our analysis, we propose an approach that periodically and efficiently optimizes the metric using standard convex constrained optimization and sketching techniques.

Author Information

Yatin Dandi (Swiss Federal Institute of Technology Lausanne)
Anastasiia Koloskova (EPFL)
Martin Jaggi (EPFL)
Sebastian Stich (CISPA)

More from the Same Authors