Timezone: »
Personalized federated learning considers learning models unique to each client in a heterogeneous network. The resulting client-specific models have been purported to improve metrics such as accuracy, fairness, and robustness in federated networks. However, despite a plethora of work in this area, it remains unclear: (1) which personalization techniques are most effective in various settings, and (2) how important personalization truly is for realistic federated applications. To better answer these questions, we propose Motley, a benchmark for personalized federated learning. Motley consists of a suite of cross-device and cross-silo federated datasets from varied problem domains, as well as thorough evaluation metrics for better understanding the possible impacts of personalization. We establish baselines on the benchmark by comparing a number of representative personalized federated learning methods. These initial results highlight strengths and weaknesses of existing approaches, and raise several open questions for the community. Motley aims to provide a reproducible means with which to advance developments in personalized and heterogeneity-aware federated learning, as well as the related areas of transfer learning, meta-learning, and multi-task learning.
Author Information
Shanshan Wu (Google)
Tian Li (CMU)
Zachary Charles (Google Research)
Yu Xiao (Google)
Ken Liu (Carnegie Mellon University)
Zheng Xu (Google Research)
Virginia Smith (Carnegie Mellon University)
More from the Same Authors
-
2021 : Iterated Vector Fields and Conservatism, with Applications to Federated Learning »
Zachary Charles · John Rush -
2021 : Efficient and Private Federated Learning with Partially Trainable Networks »
Hakim Sidahmed · Zheng Xu · Yuan Cao -
2022 : Differentially Private Adaptive Optimization with Delayed Preconditioners »
Tian Li · Manzil Zaheer · Ken Liu · Sashank Reddi · H. Brendan McMahan · Virginia Smith -
2022 : Differentially Private Adaptive Optimization with Delayed Preconditioners »
Tian Li · Manzil Zaheer · Ken Liu · Sashank Reddi · H. Brendan McMahan · Virginia Smith -
2022 : Adaptive Sparse Federated Learning in Large Output Spaces via Hashing »
Zhaozhuo Xu · Luyang Liu · Zheng Xu · Anshumali Shrivastava -
2022 : Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shifts »
Amrith Setlur · Don Dennis · Benjamin Eysenbach · Aditi Raghunathan · Chelsea Finn · Virginia Smith · Sergey Levine -
2023 Poster: (Amplified) Banded Matrix Factorization: A unified approach to private training »
Christopher Choquette-Choo · Arun Ganesh · Ryan McKenna · H. Brendan McMahan · John Rush · Abhradeep Guha Thakurta · Zheng Xu -
2023 Poster: Progressive Knowledge Distillation: Constructing Ensembles for Efficient Inference »
Don Dennis · Abhishek Shetty · Anish Prasad Sevekari · Kazuhito Koishida · Virginia Smith -
2023 Poster: Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shift »
Saurabh Garg · Amrith Setlur · Zachary Lipton · Sivaraman Balakrishnan · Virginia Smith · Aditi Raghunathan -
2023 Poster: Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacy »
Anastasiia Koloskova · Ryan McKenna · Zachary Charles · John Rush · H. Brendan McMahan -
2023 Poster: Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies »
Oscar Li · James Harrison · Jascha Sohl-Dickstein · Virginia Smith · Luke Metz -
2023 Poster: Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning »
Zachary Charles · Nicole Mitchell · Krishna Pillutla · Michael Reneer · Zachary Garrett -
2023 : Evaluating Large-Scale Learning Systems, Virginia Smith »
Virginia Smith -
2022 : Panel »
Virginia Smith · Michele Covell · Daniel Severo · Christopher Schroers -
2022 : Poster Session 1 »
Andrew Lowy · Thomas Bonnier · Yiling Xie · Guy Kornowski · Simon Schug · Seungyub Han · Nicolas Loizou · xinwei zhang · Laurent Condat · Tabea E. Röber · Si Yi Meng · Marco Mondelli · Runlong Zhou · Eshaan Nichani · Adrian Goldwaser · Rudrajit Das · Kayhan Behdin · Atish Agarwala · Mukul Gagrani · Gary Cheng · Tian Li · Haoran Sun · Hossein Taheri · Allen Liu · Siqi Zhang · Dmitrii Avdiukhin · Bradley Brown · Miaolan Xie · Junhyung Lyle Kim · Sharan Vaswani · Xinmeng Huang · Ganesh Ramachandra Kini · Angela Yuan · Weiqiang Zheng · Jiajin Li -
2022 : Contributed Talks 1 »
Courtney Paquette · Tian Li · Guy Kornowski -
2022 : To Federate or Not To Federate: Incentivizing Client Participation in Federated Learning »
Yae Jee Cho · Divyansh Jhunjhunwala · Tian Li · Virginia Smith · Gauri Joshi -
2022 Poster: On Privacy and Personalization in Cross-Silo Federated Learning »
Ken Liu · Shengyuan Hu · Steven Wu · Virginia Smith -
2022 Poster: Adversarial Unlearning: Reducing Confidence Along Adversarial Directions »
Amrith Setlur · Benjamin Eysenbach · Virginia Smith · Sergey Levine -
2021 : Contributed Talk 5: Efficient and Private Federated Learning with Partially Trainable Networks »
Hakim Sidahmed · Zheng Xu · Yuan Cao -
2021 : Q&A with A/Professor Virginia Smith »
Virginia Smith -
2021 : Keynote Talk: Fair or Robust: Addressing Competing Constraints in Federated Learning (Virginia Smith) »
Virginia Smith -
2021 Poster: Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution »
Amrith Setlur · Oscar Li · Virginia Smith -
2021 Poster: The Skellam Mechanism for Differentially Private Federated Learning »
Naman Agarwal · Peter Kairouz · Ken Liu -
2021 Poster: On Large-Cohort Training for Federated Learning »
Zachary Charles · Zachary Garrett · Zhouyuan Huo · Sergei Shmulyian · Virginia Smith -
2021 Poster: Federated Reconstruction: Partially Local Federated Learning »
Karan Singhal · Hakim Sidahmed · Zachary Garrett · Shanshan Wu · John Rush · Sushant Prakash -
2021 Poster: Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing »
Mikhail Khodak · Renbo Tu · Tian Li · Liam Li · Maria-Florina Balcan · Virginia Smith · Ameet Talwalkar -
2020 Tutorial: (Track1) Federated Learning and Analytics: Industry Meets Academia Q&A »
Peter Kairouz · Brendan McMahan · Virginia Smith -
2020 Poster: Implicit Regularization and Convergence for Weight Normalization »
Xiaoxia Wu · Edgar Dobriban · Tongzheng Ren · Shanshan Wu · Zhiyuan Li · Suriya Gunasekar · Rachel Ward · Qiang Liu -
2020 Poster: Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning »
Meng Zhou · Ken Liu · Pengwei Sui · Yixuan Li · Yuk Ying Chung -
2020 Tutorial: (Track1) Federated Learning and Analytics: Industry Meets Academia »
Brendan McMahan · Virginia Smith · Peter Kairouz -
2019 Workshop: Workshop on Federated Learning for Data Privacy and Confidentiality »
Lixin Fan · Jakub Konečný · Yang Liu · Brendan McMahan · Virginia Smith · Han Yu -
2019 Poster: Sparse Logistic Regression Learns All Discrete Pairwise Graphical Models »
Shanshan Wu · Sujay Sanghavi · Alex Dimakis -
2019 Spotlight: Sparse Logistic Regression Learns All Discrete Pairwise Graphical Models »
Shanshan Wu · Sujay Sanghavi · Alex Dimakis -
2019 Poster: DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation »
Shashank Rajput · Hongyi Wang · Zachary Charles · Dimitris Papailiopoulos -
2019 Poster: Learning Distributions Generated by One-Layer ReLU Networks »
Shanshan Wu · Alex Dimakis · Sujay Sanghavi -
2018 : Prof. Virginia Smith »
Virginia Smith -
2016 Poster: Leveraging Sparsity for Efficient Submodular Data Summarization »
Erik Lindgren · Shanshan Wu · Alex Dimakis -
2016 Poster: Single Pass PCA of Matrix Products »
Shanshan Wu · Srinadh Bhojanapalli · Sujay Sanghavi · Alex Dimakis