Timezone: »
Spotlight
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Alex Wang · Yada Pruksachatkun · Nikita Nangia · Amanpreet Singh · Julian Michael · Felix Hill · Omer Levy · Samuel Bowman
In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at https://super.gluebenchmark.com.
Author Information
Alex Wang (New York University)
Yada Pruksachatkun (New York University)
Nikita Nangia (NYU)
Amanpreet Singh (Facebook)
Julian Michael (University of Washington)
Felix Hill (Google Deepmind)
Omer Levy (Facebook AI Research)
Samuel Bowman (New York University)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems »
Thu. Dec 12th 01:00 -- 03:00 AM Room East Exhibition Hall B + C #100
More from the Same Authors
-
2022 : Two-Turn Debate Does Not Help Humans Answer Hard Reading Comprehension Questions »
Alicia Parrish · Harsh Trivedi · Nikita Nangia · Jason Phang · Vishakh Padmakumar · Amanpreet Singh Saimbhi · Samuel Bowman -
2023 Poster: Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting »
Miles Turpin · Julian Michael · Ethan Perez · Samuel Bowman -
2023 Poster: LIMA: Less Is More for Alignment »
Chunting Zhou · Pengfei Liu · Puxin Xu · Srinivasan Iyer · Jiao Sun · Yuning Mao · Xuezhe Ma · Avia Efrat · Ping Yu · LILI YU · Susan Zhang · Gargi Ghosh · Mike Lewis · Luke Zettlemoyer · Omer Levy -
2023 Poster: Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation »
Yuval Kirstain · Adam Polyak · Uriel Singer · Shahbuland Matiana · Joe Penna · Omer Levy -
2023 Workshop: Socially Responsible Language Modelling Research (SoLaR) »
Usman Anwar · David Krueger · Samuel Bowman · Jakob Foerster · Su Lin Blodgett · Roberta Raileanu · Alan Chan · Katherine Lee · Laura Ruis · Robert Kirk · Yawen Duan · Xin Chen · Kawin Ethayarajh -
2022 : Sam Bowman: What's the deal with AI safety? »
Samuel Bowman -
2022 Workshop: Human Evaluation of Generative Models »
Divyansh Kaushik · Jennifer Hsia · Jessica Huynh · Yonadav Shavit · Samuel Bowman · Ting-Hao Huang · Douwe Kiela · Zachary Lipton · Eric Michael Smith -
2021 : Invited talk 9 »
Samuel Bowman -
2021 Panel: The Role of Benchmarks in the Scientific Progress of Machine Learning »
Lora Aroyo · Samuel Bowman · Isabelle Guyon · Joaquin Vanschoren -
2020 Poster: The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes »
Douwe Kiela · Hamed Firooz · Aravind Mohan · Vedanuj Goswami · Amanpreet Singh · Pratik Ringshia · Davide Testuggine -
2019 Poster: Can Unconditional Language Models Recover Arbitrary Sentences? »
Nishant Subramani · Samuel Bowman · Kyunghyun Cho -
2019 Poster: Are Sixteen Heads Really Better than One? »
Paul Michel · Omer Levy · Graham Neubig -
2018 : Poster session »
Ralf Mayet · Paulo Orenstein · Heloise Greeff · Tomasz Rutkowski · Jiafan Yu · Milena Marin · Peter He · Jigar Doshi · Xavier Boix · Thomas Janssoone · Aniket Kesari · Yunyi Li · Arbel Vigodny · Ellie Gordon · Zach Moshe · Sella Nevo · Harvey Wu · Jessica Lee · Noel Corriveau · Vincenzo Lomonaco · Yada Pruksachatkun · Naroa Zurutuza · Bhairav Mehta · Carolyne Pelletier · Yasmeen Hitti · Sophia Latessa · Gerard Glowacki · Alexis G Gkantiragas · Oliver Nina · Íñigo Martínez de Rituerto de Troya · Vedran Sekara · Michael Madaio · Eunbee Jang · Ines Moreno · Arnon Houri-Yafin · Claire Babirye -
2014 Poster: Neural Word Embedding as Implicit Matrix Factorization »
Omer Levy · Yoav Goldberg