Timezone: »
The program includes a wide variety of exciting competitions in different domains, with some focusing more on applications and others trying to unify fields, focusing on technical challenges or directly tackling important problems in the world. The aim is for the broad program to make it so that anyone who wants to work on or learn from a competition can find something to their liking.
In this session, we have the following competitions:
* The Open Catalyst Challenge
* The Robustness and Uncertainty under Real-World Distributional Shifts Challenge
* VisDA21: Visual Domain Adaptation
* HEAR 2021: Holistic Evaluation of Audio Representations
* The WebQA Competition
* Diamond: A MineRL Competition on Training Sample-Efficient Agents
|
Tue 10:00 a.m. - 10:05 a.m.
|
Introduction Competion Day 1
(
Intro
)
|
Douwe Kiela 🔗 |
|
Tue 10:05 a.m. - 10:25 a.m.
|
Open Catalyst Challenge + Q&A
(
Talk
)
link »
SlidesLive Video » Advancements to renewable energy processes are urgently needed to address climate change and energy scarcity around the world. Many of these processes, including the generation of electricity through fuel cells or fuel generation from renewable resources are driven through chemical reactions. The design of new catalysts for enabling new and more efficient reactions is a critical bottleneck in developing cost-effective solutions. Unfortunately, the discovery of new catalyst materials is limited due to the high cost of computational atomic simulations and experimental studies. Machine learning has the potential to significantly reduce the cost of computational simulations by orders of magnitude. By filtering potential catalyst materials based on these simulations, candidates of higher promise may be selected for experimental testing and the rate at which new catalysts are discovered could be greatly increased. The Open Catalyst Challenge invites participants to submit results of machine learning models that simulate the interaction of a molecule on a catalyst's surface. ML models may either directly predict the relaxed state atomic configuration of the entire molecule + catalyst system, or iteratively predict and integrate per-atom forces to simulate how atoms will move around starting from an arbitrary initial state. By predicting this interaction accurately, the catalyst's impact on the overall rate of a chemical reaction may be estimated; a key factor in filtering potential catalysis materials and addressing the world’s energy needs. The Open Catalyst Project is a collaborative research effort between Facebook AI Research (FAIR) and Carnegie Mellon University’s (CMU) Department of Chemical Engineering. The aim is to use AI to model and discover new catalysts for use in renewable energy storage to help in addressing climate change. |
Abhishek Das · Muhammed Shuaibi · Siddharth Goyal · Adeesh Kolluru · Janice Lan · Aini Palizhati · Anuroop Sriram · Brandon Wood · Aditya Grover · Devi Parikh · Zachary Ulissi · Larry Zitnick
|
|
Tue 10:24 a.m. - 1:24 p.m.
|
Breakout: Open Catalyst Challenge Breakout Session
(
Breakout session
)
Schedule (GMT TImezone)
|
🔗 |
|
Tue 10:25 a.m. - 10:45 a.m.
|
Shifts Challenge: Robustness and Uncertainty under Real-World Distributional Shift + Q&A
(
Talk
)
link »
SlidesLive Video » Mismatch between training and deployment data, known as distributional shift, adversely impacts ML models and is ubiquitous in real, industrial applications. In this competition the contestants’ goal is to develop models which are both robust to distributional shift and can detect it via uncertainty estimation. The broad aim of this competition is to raise awareness of the issue and stimulate the community to work on tasks and modalities taken from large-scale industrial applications. Thus, we provide the "Shifts Dataset" - a new, large dataset of genuine `in the wild' examples of distributional shift from weather prediction, machine translation, and vehicle motion prediction. Each task represents a particular data-modality and is uniquely challenging. Each task will have an associated competition track with prizes for top contestants. |
Andrey Malinin · Neil Band · German Chesnokov · Yarin Gal · Alexander Ganshin · Mark Gales · Alexey Noskov · Liudmila Prokhorenkova · Mariya Shmatova · Vyas Raina · Vatsal Raina · Panagiotis Tigas · Boris Yangel
|
|
Tue 10:44 a.m. - 1:44 p.m.
|
Breakout: Shifts Challenge: Robustness and Uncertainty under Real-World Distributional Shift
(
Breakout session
)
Schedule (GMT Timezone)
|
🔗 |
|
Tue 10:45 a.m. - 11:05 a.m.
|
VisDA21: Visual Domain Adaptation + Q&A
(
Talk
)
link »
SlidesLive Video » Progress in machine learning is typically measured by training and testing a model on the same distribution of data, i.e., the same domain. However, in real world applications, models often encounter out-of-distribution data. The VisDA21 competition invites methods that can adapt to novel test distributions and handle distributional shifts. Our task is object classification, but we measure accuracy on novel domains, rather than the traditional in-domain benchmarking. Teams will be given labeled source data and unlabeled target data from a different distribution (such as novel viewpoints, backgrounds, image quality). In addition, the target data may have missing and/or novel classes. Successful approaches will improve classification accuracy of known categories on target-domain data while learning to deal with missing and/or unknown categories. |
Kate Saenko · Kuniaki Saito · Donghyun Kim · Samarth Mishra · Ben Usman · Piotr Teterwak · Dina Bashkirova · Dan Hendrycks 🔗 |
|
Tue 11:04 a.m. - 2:04 p.m.
|
Breakout: VisDA21: Visual Domain Adaptation
(
Break out session
)
|
🔗 |
|
Tue 11:05 a.m. - 11:25 a.m.
|
HEAR 2021: Holistic Evaluation of Audio Representations + Q&A
(
Talk
)
link »
SlidesLive Video » Humans can infer a wide range of properties from a perceived sound, such as information about the source (e.g. what generated the sound? where is it coming from?), the information the sound conveys (this is a word that means X, this is a musical note in scale Y), and how it compares to other sounds (these two sounds come/don't come from the same source and are/aren't identical). Can any one learned representation do the same? The aim of this competition is to develop a general-purpose audio representation that provides a meaningful basis for learning in a wide variety of tasks and scenarios. We challenge participants with the following questions: Is it possible to develop a single representation that models all psychoacoustic phenomena? What approach best generalizes to a wide range of downstream audio tasks without fine-tuning? What audio representation allows researchers to formulate and solve novel and societally-valuable problems in simple, repeatable ways? We will evaluate audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, medical audio, and music. In the spirit of shared exchange, all participants must submit an audio embedding model, following a common API, that is general-purpose, open-source, and freely available to use. |
Joseph Turian · Jordan Shier · Bhiksha Raj · Bjoern Schuller · Christian Steinmetz · George Tzanetakis · Gissel Velarde · Kirk McNally · Max Henry · Nicolas Pinto · Yonatan Bisk · George Tzanetakis · Camille Noufi · Dorien Herremans · Jesse Engel · Justin Salamon · Prany Manocha · Philippe Esling · Shinji Watanabe
|
|
Tue 11:24 a.m. - 2:24 p.m.
|
Breakout: HEAR 2021: Holistic Evaluation of Audio Representations
(
Breakout session
)
Schedule (GMT Timezone) Invited speakers have been asked to prepare a 5-minute lightning talk. Following the lightning talks we will have an open question and discussion period. The breakout session will start at 19:25 GMT and last for approximately one hour. The current tentative team order for lightning talks is:
|
🔗 |
|
Tue 11:25 a.m. - 11:45 a.m.
|
WebQA Competition + Q&A
(
Talk
)
link »
SlidesLive Video » WebQA is a new benchmark for multimodal multihop reasoning in which systems are presented with the same style of data as humans when searching the web: snippets and images. Upon seeing a question, the system must identify which candidates potentially inform the answer from a candidate pool. Then the system is expected to aggregate information from selected candidates with reasoning to generate an answer in natural language form. Each datum is a question paired with a series of potentially long snippets or images that serve as "knowledge carriers" over which to reason. Systems will be evaluated on both supporting fact retrieval and answer generation to measure correctness and interpretability. To demonstrate multihop multimodal reasoning ability, models should be able to 1) understand and represent knowledge from different modalities, 2) identify and aggregate relevant knowledge fragments scattered across multiple sources, 3) make inference and do natural language generation. |
Yingshan CHANG · Yonatan Bisk · Mridu Narang · Levi Melnick · Jianfeng Gao · Hisami Suzuki · Guihong Cao 🔗 |
|
Tue 11:44 a.m. - 2:44 p.m.
|
Breakout: WebQA Competition
(
Breakout session
)
|
🔗 |
|
Tue 11:45 a.m. - 12:05 p.m.
|
Diamond: A MineRL Competition on Training Sample-Efficient Agents + Q&A
(
Talk
)
link »
SlidesLive Video » In the third MineRL Diamond competition, participants continue to develop algorithms which can efficiently leverage human demonstrations to drastically reduce the number of samples needed to solve a complex task in Minecraft. The competition environment features sparse-rewards, long-term planning, vision and sub-task hierarchies. To ensure that truly sample-efficient are developed, organizers re-train submitted systems on a fixed cloud-computing environment for a limited number of samples (4 days or 8 million samples). To ease the entry to machine learning research, the competition features two tracks: introduction, which allows agents developed using any method ranging from end-to-end machine learning solutions to programmatic approaches; and research, which requires participants develop novel imitation and reinforcement learning algorithms to solve this difficult sample-limited task. |
William Guss · Alara Dirik · Byron Galbraith · Brandon Houghton · Anssi Kanervisto · Noboru Kuno · Stephanie Milani · Sharada Mohanty · Karolis Ramanauskas · Ruslan Salakhutdinov · Rohin Shah · Nicholay Topin · Steven Wang · Cody Wild
|
|
Tue 12:04 p.m. - 3:04 p.m.
|
Breakout: Diamond: A MineRL Competition on Training Sample-Efficient Agents
(
Breakout session
)
|
🔗 |
Author Information
Douwe Kiela (Facebook AI Research)
Marco Ciccone (Politecnico di Torino)
Marco Ciccone is an ELLIS Postdoctoral Researcher in the VANDAL group at Politecnico di Torino and UCL. His current research interests are in the intersection of meta, continual, and federated learning with a particular focus on modularity and models re-use to scale the training of agents with heterogeneous data and mitigate the effect of catastrophic forgetting and interference across tasks, domains, and devices. He has been NeurIPS Competiton Track co-chair in 2021, 2022 and 2023.
Barbara Caputo (Politecnico di Torino)
More from the Same Authors
-
2021 : Public Information Representation for Adversarial Team Games »
Luca Carminati · Federico Cacciamani · Marco Ciccone · Nicola Gatti -
2022 : Perturbation Augmentation for Fairer NLP »
Rebecca Qian · Candace Ross · Jude Fernandes · Eric Michael Smith · Douwe Kiela · Adina Williams -
2023 Poster: OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents »
Hugo Laurençon · Lucile Saulnier · Leo Tronchon · Stas Bekman · Amanpreet Singh · Anton Lozhkov · Thomas Wang · Siddharth Karamcheti · Alexander Rush · Douwe Kiela · Matthieu Cord · Victor Sanh -
2023 Poster: DataPerf: Benchmarks for Data-Centric AI Development »
Mark Mazumder · Colby Banbury · Xiaozhe Yao · Bojan Karlaš · William Gaviria Rojas · Sudnya Diamos · Greg Diamos · Lynn He · Alicia Parrish · Hannah Rose Kirk · Jessica Quaye · Charvi Rastogi · Douwe Kiela · David Jurado · David Kanter · Rafael Mosquera · Will Cukierski · Juan Ciro · Lora Aroyo · Bilge Acun · Lingjiao Chen · Mehul Raje · Max Bartolo · Evan Sabri Eyuboglu · Amirata Ghorbani · Emmett Goodman · Addison Howard · Oana Inel · Tariq Kane · Christine R. Kirkpatrick · D. Sculley · Tzu-Sheng Kuo · Jonas Mueller · Tristan Thrush · Joaquin Vanschoren · Margaret Warren · Adina Williams · Serena Yeung · Newsha Ardalani · Praveen Paritosh · Ce Zhang · James Zou · Carole-Jean Wu · Cody Coleman · Andrew Ng · Peter Mattson · Vijay Janapa Reddi -
2022 Workshop: Human Evaluation of Generative Models »
Divyansh Kaushik · Jennifer Hsia · Jessica Huynh · Yonadav Shavit · Samuel Bowman · Ting-Hao Huang · Douwe Kiela · Zachary Lipton · Eric Michael Smith -
2022 Competition: NeurIPS 2022 Competition Track: Overview & Results »
Marco Ciccone · Gustavo Stolovitzky · Jake Albrecht -
2021 : Spotlight Talk: Public Information Representation for Adversarial Team Games »
Luca Carminati · Federico Cacciamani · Marco Ciccone · Nicola Gatti -
2021 : Facebook - Data Centric Infrastructure »
Douwe Kiela -
2021 Demonstration: Demonstrations 4 »
Douwe Kiela · Barbara Caputo · Marco Ciccone -
2021 : Intro »
Marco Ciccone -
2021 : Introduction to Competition Day 4 »
Marco Ciccone -
2021 Competition: Competition Track Day 4: Overviews + Breakout Sessions »
Douwe Kiela · Marco Ciccone · Barbara Caputo -
2021 Poster: True Few-Shot Learning with Language Models »
Ethan Perez · Douwe Kiela · Kyunghyun Cho -
2021 : Invited talk - Douwe Kiela »
Douwe Kiela -
2021 : Introduction to Competition Day 3 »
Marco Ciccone -
2021 Competition: Competition Track Day 3: Overviews + Breakout Sessions »
Douwe Kiela · Marco Ciccone · Barbara Caputo -
2021 Demonstration: Demonstrations 3 »
Douwe Kiela · Barbara Caputo · Marco Ciccone -
2021 : Intro »
Marco Ciccone -
2021 Poster: Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking »
Zhiyi Ma · Kawin Ethayarajh · Tristan Thrush · Somya Jain · Ledell Wu · Robin Jia · Christopher Potts · Adina Williams · Douwe Kiela -
2021 Demonstration: Demonstrations 2 »
Douwe Kiela · Barbara Caputo · Marco Ciccone -
2021 : Intro »
Douwe Kiela -
2021 : Introduction to Competition Day 2 »
Barbara Caputo -
2021 Competition: Competition Track Day 2: Overviews + Breakout Sessions »
Douwe Kiela · Marco Ciccone · Barbara Caputo -
2021 : Introduction Competion Day 1 »
Douwe Kiela -
2021 Poster: Human-Adversarial Visual Question Answering »
Sasha Sheng · Amanpreet Singh · Vedanuj Goswami · Jose Magana · Tristan Thrush · Wojciech Galuba · Devi Parikh · Douwe Kiela -
2021 Demonstration: Demonstrations 1 »
Douwe Kiela · Barbara Caputo · Marco Ciccone -
2021 : Introduction »
Douwe Kiela -
2020 : Q & A and Panel Session with Dan Weld, Kristen Grauman, Scott Yih, Emma Brunskill, and Alex Ratner »
Kristen Grauman · Wen-tau Yih · Alexander Ratner · Emma Brunskill · Douwe Kiela · Daniel S. Weld -
2020 Workshop: HAMLETS: Human And Model in the Loop Evaluation and Training Strategies »
Divyansh Kaushik · Bhargavi Paranjape · Forough Arabshahi · Yanai Elazar · Yixin Nie · Max Bartolo · Polina Kirichenko · Pontus Lars Erik Saito Stenetorp · Mohit Bansal · Zachary Lipton · Douwe Kiela -
2020 : Opening Remarks »
Divyansh Kaushik · Bhargavi Paranjape · Douwe Kiela -
2020 : The Hateful Memes Challenge: Live award ceremony and winner presentations »
Douwe Kiela -
2020 : The Hateful Memes Challenge: Competition Overview »
Douwe Kiela -
2020 Poster: The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes »
Douwe Kiela · Hamed Firooz · Aravind Mohan · Vedanuj Goswami · Amanpreet Singh · Pratik Ringshia · Davide Testuggine -
2020 Poster: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks »
Patrick Lewis · Ethan Perez · Aleksandra Piktus · Fabio Petroni · Vladimir Karpukhin · Naman Goyal · Heinrich Küttler · Mike Lewis · Wen-tau Yih · Tim Rocktäschel · Sebastian Riedel · Douwe Kiela -
2020 Poster: Learning Optimal Representations with the Decodable Information Bottleneck »
Yann Dubois · Douwe Kiela · David Schwab · Ramakrishna Vedantam -
2020 Spotlight: Learning Optimal Representations with the Decodable Information Bottleneck »
Yann Dubois · Douwe Kiela · David Schwab · Ramakrishna Vedantam -
2019 : Audrey Durand, Douwe Kiela, Kamalika Chaudhuri moderated by Yann Dauphin »
Audrey Durand · Kamalika Chaudhuri · Yann Dauphin · Orhan Firat · Dilan Gorur · Douwe Kiela -
2019 : Douwe Kiela - Benchmarking Progress in AI: A New Benchmark for Natural Language Understanding »
Douwe Kiela -
2019 Workshop: Emergent Communication: Towards Natural Language »
Abhinav Gupta · Michael Noukhovitch · Cinjon Resnick · Natasha Jaques · Angelos Filos · Marie Ossenkopf · Angeliki Lazaridou · Jakob Foerster · Ryan Lowe · Douwe Kiela · Kyunghyun Cho -
2019 Poster: Hyperbolic Graph Neural Networks »
Qi Liu · Maximilian Nickel · Douwe Kiela -
2018 Workshop: Emergent Communication Workshop »
Jakob Foerster · Angeliki Lazaridou · Ryan Lowe · Igor Mordatch · Douwe Kiela · Kyunghyun Cho -
2018 : Panel Discussion »
Antonio Torralba · Douwe Kiela · Barbara Landau · Angeliki Lazaridou · Joyce Chai · Christopher Manning · Stevan Harnad · Roozbeh Mottaghi -
2018 : Douwe Kiela - Learning Multimodal Embeddings »
Douwe Kiela -
2018 Poster: NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations »
Marco Ciccone · Marco Gallieri · Jonathan Masci · Christian Osendorfer · Faustino Gomez -
2017 Workshop: Emergent Communication Workshop »
Jakob Foerster · Igor Mordatch · Angeliki Lazaridou · Kyunghyun Cho · Douwe Kiela · Pieter Abbeel -
2017 Poster: Poincaré Embeddings for Learning Hierarchical Representations »
Maximilian Nickel · Douwe Kiela -
2017 Spotlight: Poincaré Embeddings for Learning Hierarchical Representations »
Maximilian Nickel · Douwe Kiela -
2009 Workshop: Learning from Multiple Sources with Applications to Robotics »
Barbara Caputo · Nicolò Cesa-Bianchi · David R Hardoon · Gayle Leen · Francesco Orabona · Jaakko Peltonen · Simon Rogers -
2009 Poster: Who’s Doing What: Joint Modeling of Names and Verbs for Simultaneous Face and Pose Annotation »
Jie Luo · Barbara Caputo · Vittorio Ferrari