Timezone: »

 
Oral
Scaling Data-Constrained Language Models
Niklas Muennighoff · Alexander Rush · Boaz Barak · Teven Le Scao · Nouamane Tazi · Aleksandra Piktus · Sampo Pyysalo · Thomas Wolf · Colin Raffel

Tue Dec 12 02:10 PM -- 02:25 PM (PST) @
in Oral 2A »
Event URL: https://github.com/huggingface/datablations »

The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.

Author Information

Niklas Muennighoff (Hugging Face)
Alexander Rush (Cornell University)
Alexander Rush

Alexander "Sasha" Rush is an Associate Professor at Cornell Tech and a researcher at Hugging Face. His research interest is in the study of language models with applications in controllable text generation, efficient inference, and applications in summarization and information extraction. In addition to research, he has written several popular open-source software projects supporting NLP research, programming for deep learning, and virtual academic conferences. His projects have received paper and demo awards at major NLP, visualization, and hardware conferences, an NSF Career Award and Sloan Fellowship. He tweets at @srush_nlp.

Boaz Barak (Harvard University)
Teven Le Scao (Mistral)
Nouamane Tazi (Hugging Face)
Aleksandra Piktus (Cohere)
Sampo Pyysalo (University of Turku)
Thomas Wolf (HuggingFace Inc.)
Colin Raffel (UNC Chapel Hill and Hugging Face)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors