Timezone: »

 
Motley: Benchmarking Heterogeneity and Personalization in Federated Learning
Shanshan Wu · Tian Li · Zachary Charles · Yu Xiao · Ken Liu · Zheng Xu · Virginia Smith

@
in Federated Learning: Recent Advances and New Challenges »
Event URL: https://openreview.net/forum?id=7vTCrWhNU_D »

Personalized federated learning considers learning models unique to each client in a heterogeneous network. The resulting client-specific models have been purported to improve metrics such as accuracy, fairness, and robustness in federated networks. However, despite a plethora of work in this area, it remains unclear: (1) which personalization techniques are most effective in various settings, and (2) how important personalization truly is for realistic federated applications. To better answer these questions, we propose Motley, a benchmark for personalized federated learning. Motley consists of a suite of cross-device and cross-silo federated datasets from varied problem domains, as well as thorough evaluation metrics for better understanding the possible impacts of personalization. We establish baselines on the benchmark by comparing a number of representative personalized federated learning methods. These initial results highlight strengths and weaknesses of existing approaches, and raise several open questions for the community. Motley aims to provide a reproducible means with which to advance developments in personalized and heterogeneity-aware federated learning, as well as the related areas of transfer learning, meta-learning, and multi-task learning.

Author Information

Shanshan Wu (Google)
Tian Li (CMU)
Zachary Charles (Google Research)
Yu Xiao (Google)
Ken Liu (Carnegie Mellon University)
Zheng Xu (Google Research)
Virginia Smith (Carnegie Mellon University)

More from the Same Authors