Timezone: »

 
Training Transformers Together
Alexander Borzunov · Max Ryabinin · Tim Dettmers · quentin lhoest · Lucile Saulnier · Michael Diskin · Yacine Jernite · Thomas Wolf

Tue Dec 07 09:05 AM -- 09:20 AM (PST) @
in Demonstrations 1 »
Event URL: https://training-transformers-together.github.io/ »

We invite volunteers to train a large Transformer language model over the Internet. Instead of using supercomputers, we will pool together all available computational resources: desktops, laptops, servers and even cloud TPUs from around the world. All training artifacts, such as model checkpoint and optimizer states, will be shared online for public use.

For this demonstration, we will provide an open-source starter kit that volunteers can use to join the global distributed training run and host similar experiments independently in the future.

Author Information

Alexander Borzunov (HSE University, Yandex)
Max Ryabinin (Yandex, Higher School of Economics)
Tim Dettmers (University of Washington)
quentin lhoest (Hugging Face)
Lucile Saulnier (Hugging Face)
Michael Diskin (Yandex, Higher School of Economics)
Yacine Jernite (Facebook FAIR NYC)
Thomas Wolf (🤗 Hugging Face)

More from the Same Authors