Timezone: »

 
Poster
Distributed Deep Learning In Open Collaborations
Michael Diskin · Alexey Bukhtiyarov · Max Ryabinin · Lucile Saulnier · quentin lhoest · Anton Sinitsin · Dmitry Popov · Dmitry V. Pyrkin · Maxim Kashirin · Alexander Borzunov · Albert Villanova del Moral · Denis Mazur · Ilia Kobelev · Yacine Jernite · Thomas Wolf · Gennady Pekhimenko

Fri Dec 10 08:30 AM -- 10:00 AM (PST) @ Virtual
in Poster Session 8 »

Modern deep learning applications require increasingly more compute to train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High-Performance Computing clusters, whose construction and maintenance are both environmentally costly and well beyond the budget of most organizations. As a result, some research directions become the exclusive domain of a few large industrial and even fewer academic actors. To alleviate this disparity, smaller groups may pool their computational resources and run collaborative experiments that benefit all participants. This paradigm, known as grid- or volunteer computing, has seen successful applications in numerous scientific areas. However, using this approach for machine learning is difficult due to high latency, asymmetric bandwidth, and several challenges unique to volunteer computing. In this work, we carefully analyze these constraints and propose a novel algorithmic framework designed specifically for collaborative training. We demonstrate the effectiveness of our approach for SwAV and ALBERT pretraining in realistic conditions and achieve performance comparable to traditional setups at a fraction of the cost. Finally, we provide a detailed report of successful collaborative language model pretraining with nearly 50 participants.

Keywords: Deep Learning · Machine Learning · Generative Model · Transfer Learning

Author Information

Michael Diskin (Yandex, Higher School of Economics)
Alexey Bukhtiyarov (Moscow Institute of Physics and Technology)
Max Ryabinin (Yandex, Higher School of Economics)
Lucile Saulnier (Hugging Face)
quentin lhoest (Hugging Face)
Anton Sinitsin (Yandex)

Software developer in Yandex, Dl researcher

Dmitry Popov (Higher School of Economics)
Dmitry V. Pyrkin (National Research University Higher School of Economics)
Maxim Kashirin (Higher School of Economics)
Alexander Borzunov (HSE University, Yandex)
Albert Villanova del Moral (CNRS)
Denis Mazur (Yandex)

Badge Text

Ilia Kobelev (Moscow Institute of Physics and Technology)
Yacine Jernite (Hugging Face)
Thomas Wolf (🤗 Hugging Face)
Gennady Pekhimenko (University of Toronto)

More from the Same Authors