On Communication-Efficient Training of Ensembles in Federated Learning

In recent years, ensemble ideas have found parallels in federated learning (FL). Specific ensemble learning formulations under the federated setup involve information exchange between clients during the training process, making the classic FL communication bottleneck a critical issue. Importantly, the resulting communication patterns differ from those studied in traditional communication-efficient FL methods, placing ensembling outside their analytical scope. In this work, we close this gap by extending classic communication-efficient techniques to the ensembling setting and introducing a new mechanism specifically tailored to it. We instantiate these ideas in four algorithms and provide convergence guarantees for each under the smooth non-convex setup. We further support our theoretical findings with experimental results.