Timezone: »

 
Understanding Federated Learning through Loss Landscape Visualizations: A Pilot Study
Ziwei Li · Hong-You Chen · Han Wei Shen · Wei-Lun (Harry) Chao

@
in Federated Learning: Recent Advances and New Challenges »
Event URL: https://openreview.net/forum?id=nUzVXJ02bSe »

Federated learning aims to train a machine learning model (e.g., a neural network) in a data-decentralized fashion. The key challenge is the potential data heterogeneity among clients. When clients' data are non-IID, federatedly learned models could hardly achieve the same performance as centralizedly learned models. In this paper, we conduct the very first, pilot study to understand the challenge of federated learning through the lens of loss landscapes. We extend the visualization methods developed to uncover the training trajectory of centralized learning to federated learning, and explore the effect of data heterogeneity on model training. Through our approach, we can clearly visualize the phenomenon of model drifting: the more the data heterogeneity is, the larger the model drifting is. We further explore how model initialization affects the loss landscape, and how clients' participation affects the model training trajectory. We expect our approach to serve as a new, qualitative way to analyze federated learning.

Author Information

Ziwei Li (The Ohio State University)
Hong-You Chen (The Ohio State University)
Han Wei Shen (Ohio State University, Columbus)
Wei-Lun (Harry) Chao (Ohio State University (OSU))

More from the Same Authors