Timezone: »

 
Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout
Chen Dun · Mirian Hipolito Garcia · Dimitrios Dimitriadis · Christopher Jermaine · Anastasios Kyrillidis

@
in Federated Learning: Recent Advances and New Challenges »
Event URL: https://openreview.net/forum?id=DNzb6IdvoUX »

We focus on dropout techniques for asynchronous distributed computations in federated learning (FL) scenarios. We propose \texttt{AsyncDrop}, a novel asynchronous FL framework with smart (i.e., informed/structured) dropout that achieves better performance compared to state of the art asynchronous methodologies, while resulting in less communication and training time costs. The key idea revolves around sub-models out of the global model, that take into account the device heterogeneity. We conjecture that such an approach can be theoretically justified. We implement our approach and compare it against other asynchronous baseline methods, by adapting current synchronous FL algorithms to asynchronous scenarios. Empirically, \texttt{AsyncDrop} significantly reduces the communication cost and training time, while improving the final test accuracy in non-i.i.d. scenarios.

Author Information

Chen Dun (Rice University)
Mirian Hipolito Garcia (Research, Microsoft)

Mirian Hipólito holds a Bachelor’s degree in Digital Systems and Robotics Engineering from Tecnológico de Monterrey in 2020. She previously worked in the automotive sector, gaining experience in the fields of autonomous vehicles, robotics and automation. She currently works as a Research Software engineer in MSR as part of Privacy in Artificial Intelligence Team.

Dimitrios Dimitriadis (Microsoft Research)
Christopher Jermaine (William Marsh Rice University)
Anastasios Kyrillidis (Rice University)

More from the Same Authors