Timezone: »
Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contrasts with the current static language modelling paradigm, which trains and evaluates models on utterances from overlapping time periods. Despite impressive recent progress, we demonstrate that Transformer-XL language models perform worse in the realistic setup of predicting future utterances from beyond their training period, and that model performance becomes increasingly worse with time. We find that, while increasing model size alone—a key driver behind recent progress—does not solve this problem, having models that continually update their knowledge with new information can indeed mitigate this performance degradation over time. Hence, given the compilation of ever-larger language modelling datasets, combined with the growing list of language-model-based NLP applications that require up-to-date factual knowledge about the world, we argue that now is the right time to rethink the static way in which we currently train and evaluate our language models, and develop adaptive language models that can remain up-to-date with respect to our ever-changing and non-stationary world. We publicly release our dynamic, streaming language modelling benchmarks for WMT and arXiv to facilitate language model evaluation that takes temporal dynamics into account.
Author Information
Angeliki Lazaridou (DeepMind)
Adhi Kuncoro (University of Oxford & DeepMind)
Elena Gribovskaya (Deepmind Google)
Devang Agrawal (DeepMind)
Adam Liska (DeepMind)
Tayfun Terzi (DeepMind)
Mai Gimenez
Cyprien de Masson d'Autume (Google DeepMind)
Tomas Kocisky (DeepMind)
Sebastian Ruder (DeepMind)
Dani Yogatama (Google DeepMind)
Kris Cao (DeepMind)
Susannah Barlow (DeepMind)
Phil Blunsom (DeepMind and Oxford University)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Spotlight: Mind the Gap: Assessing Temporal Generalization in Neural Language Models »
Dates n/a. Room
More from the Same Authors
-
2021 : LiRo: Benchmark and leaderboard for Romanian language tasks »
Stefan Dumitrescu · Petru Rebeja · Beata Lorincz · Mihaela Gaman · Andrei Avram · Mihai Ilie · Andrei Pruteanu · Adriana Stan · Lorena Rosia · Cristina Iacobescu · Luciana Morogan · George Dima · Gabriel Marchidan · Traian Rebedea · Madalina Chitez · Dani Yogatama · Sebastian Ruder · Radu Tudor Ionescu · Razvan Pascanu · Viorica Patraucean -
2023 Poster: Intriguing Properties of Quantization at Scale »
Arash Ahmadian · Saurabh Dash · Hongyu Chen · Bharat Venkitesh · Zhen Stephen Gou · Phil Blunsom · Ahmet Üstün · Sara Hooker -
2022 Spotlight: A Contrastive Framework for Neural Text Generation »
Yixuan Su · Tian Lan · Yan Wang · Dani Yogatama · Lingpeng Kong · Nigel Collier -
2022 Workshop: Transfer Learning for Natural Language Processing »
Alon Albalak · Colin Raffel · Chunting Zhou · Deepak Ramachandran · Xuezhe Ma · Sebastian Ruder -
2022 Poster: A Contrastive Framework for Neural Text Generation »
Yixuan Su · Tian Lan · Yan Wang · Dani Yogatama · Lingpeng Kong · Nigel Collier -
2021 Poster: End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering »
Devendra Singh · Siva Reddy · Will Hamilton · Chris Dyer · Dani Yogatama -
2021 Poster: Compacter: Efficient Low-Rank Hypercomplex Adapter Layers »
Rabeeh Karimi Mahabadi · James Henderson · Sebastian Ruder -
2021 Poster: Dynamic population-based meta-learning for multi-agent communication with natural language »
Abhinav Gupta · Marc Lanctot · Angeliki Lazaridou -
2020 Workshop: Talking to Strangers: Zero-Shot Emergent Communication »
Marie Ossenkopf · Angelos Filos · Abhinav Gupta · Michael Noukhovitch · Angeliki Lazaridou · Jakob Foerster · Kalesha Bullard · Rahma Chaabouni · Eugene Kharitonov · Roberto Dessì -
2019 Workshop: Emergent Communication: Towards Natural Language »
Abhinav Gupta · Michael Noukhovitch · Cinjon Resnick · Natasha Jaques · Angelos Filos · Marie Ossenkopf · Angeliki Lazaridou · Jakob Foerster · Ryan Lowe · Douwe Kiela · Kyunghyun Cho -
2019 Poster: Episodic Memory in Lifelong Language Learning »
Cyprien de Masson d'Autume · Sebastian Ruder · Lingpeng Kong · Dani Yogatama -
2019 Poster: Training Language GANs from Scratch »
Cyprien de Masson d'Autume · Shakir Mohamed · Mihaela Rosca · Jack Rae -
2019 Poster: Biases for Emergent Communication in Multi-agent Reinforcement Learning »
Tom Eccles · Yoram Bachrach · Guy Lever · Angeliki Lazaridou · Thore Graepel -
2018 Workshop: Emergent Communication Workshop »
Jakob Foerster · Angeliki Lazaridou · Ryan Lowe · Igor Mordatch · Douwe Kiela · Kyunghyun Cho -
2018 Workshop: Wordplay: Reinforcement and Language Learning in Text-based Games »
Adam Trischler · Angeliki Lazaridou · Yonatan Bisk · Wendy Tay · Nate Kushman · Marc-Alexandre Côté · Alessandro Sordoni · Daniel Ricks · Tom Zahavy · Hal Daumé III -
2018 : Panel Discussion »
Antonio Torralba · Douwe Kiela · Barbara Landau · Angeliki Lazaridou · Joyce Chai · Christopher Manning · Stevan Harnad · Roozbeh Mottaghi -
2018 : Angeliki Lazaridou - Emergence of (linguistic communication) through multi-agent interactions »
Angeliki Lazaridou -
2018 Poster: Neural Arithmetic Logic Units »
Andrew Trask · Felix Hill · Scott Reed · Jack Rae · Chris Dyer · Phil Blunsom -
2017 Workshop: Emergent Communication Workshop »
Jakob Foerster · Igor Mordatch · Angeliki Lazaridou · Kyunghyun Cho · Douwe Kiela · Pieter Abbeel -
2017 Poster: A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning »
Marc Lanctot · Vinicius Zambaldi · Audrunas Gruslys · Angeliki Lazaridou · Karl Tuyls · Julien Perolat · David Silver · Thore Graepel -
2016 : Multi-Agent Communication and the Emergence of (Natural) Language »
Angeliki Lazaridou -
2016 Workshop: Machine Intelligence @ NIPS »
Tomas Mikolov · Baroni Marco · Armand Joulin · Germán Kruszewski · Angeliki Lazaridou · Klemen Simonic -
2015 Poster: Teaching Machines to Read and Comprehend »
Karl Moritz Hermann · Tomas Kocisky · Edward Grefenstette · Lasse Espeholt · Will Kay · Mustafa Suleyman · Phil Blunsom -
2015 Poster: Learning to Transduce with Unbounded Memory »
Edward Grefenstette · Karl Moritz Hermann · Mustafa Suleyman · Phil Blunsom