The program includes a wide variety of exciting competitions in different domains, with some focusing more on applications and others trying to unify fields, focusing on technical challenges or directly tackling important problems in the world. The aim is for the broad program to make it so that anyone who wants to work on or learn from a competition can find something to their liking.
In this session, we have the following competitions:
* The Open Catalyst Challenge
* The Robustness and Uncertainty under Real-World Distributional Shifts Challenge
* VisDA21: Visual Domain Adaptation
* HEAR 2021: Holistic Evaluation of Audio Representations
* The WebQA Competition
* Diamond: A MineRL Competition on Training Sample-Efficient Agents
Tue 10:00 a.m. - 10:05 a.m.
|
Introduction Competion Day 1
(
Intro
)
|
Douwe Kiela 🔗 |
Tue 10:05 a.m. - 10:25 a.m.
|
Open Catalyst Challenge + Q&A
(
Talk
)
link »
SlidesLive Video » Advancements to renewable energy processes are urgently needed to address climate change and energy scarcity around the world. Many of these processes, including the generation of electricity through fuel cells or fuel generation from renewable resources are driven through chemical reactions. The design of new catalysts for enabling new and more efficient reactions is a critical bottleneck in developing cost-effective solutions. Unfortunately, the discovery of new catalyst materials is limited due to the high cost of computational atomic simulations and experimental studies. Machine learning has the potential to significantly reduce the cost of computational simulations by orders of magnitude. By filtering potential catalyst materials based on these simulations, candidates of higher promise may be selected for experimental testing and the rate at which new catalysts are discovered could be greatly increased. The Open Catalyst Challenge invites participants to submit results of machine learning models that simulate the interaction of a molecule on a catalyst's surface. ML models may either directly predict the relaxed state atomic configuration of the entire molecule + catalyst system, or iteratively predict and integrate per-atom forces to simulate how atoms will move around starting from an arbitrary initial state. By predicting this interaction accurately, the catalyst's impact on the overall rate of a chemical reaction may be estimated; a key factor in filtering potential catalysis materials and addressing the world’s energy needs. The Open Catalyst Project is a collaborative research effort between Facebook AI Research (FAIR) and Carnegie Mellon University’s (CMU) Department of Chemical Engineering. The aim is to use AI to model and discover new catalysts for use in renewable energy storage to help in addressing climate change. |
Abhishek Das · Muhammed Shuaibi · Siddharth Goyal · Adeesh Kolluru · Janice Lan · Aini Palizhati · Anuroop Sriram · Brandon Wood · Aditya Grover · Devi Parikh · Zachary Ulissi · Larry Zitnick
|
Tue 10:24 a.m. - 1:24 p.m.
|
Breakout: Open Catalyst Challenge Breakout Session
(
Breakout session
)
Schedule (GMT TImezone)
|
🔗 |
Tue 10:25 a.m. - 10:45 a.m.
|
Shifts Challenge: Robustness and Uncertainty under Real-World Distributional Shift + Q&A
(
Talk
)
link »
SlidesLive Video » Mismatch between training and deployment data, known as distributional shift, adversely impacts ML models and is ubiquitous in real, industrial applications. In this competition the contestants’ goal is to develop models which are both robust to distributional shift and can detect it via uncertainty estimation. The broad aim of this competition is to raise awareness of the issue and stimulate the community to work on tasks and modalities taken from large-scale industrial applications. Thus, we provide the "Shifts Dataset" - a new, large dataset of genuine `in the wild' examples of distributional shift from weather prediction, machine translation, and vehicle motion prediction. Each task represents a particular data-modality and is uniquely challenging. Each task will have an associated competition track with prizes for top contestants. |
Andrey Malinin · Neil Band · German Chesnokov · Yarin Gal · Alexander Ganshin · Mark Gales · Alexey Noskov · Liudmila Prokhorenkova · Mariya Shmatova · Vyas Raina · Vatsal Raina · Panagiotis Tigas · Boris Yangel
|
Tue 10:44 a.m. - 1:44 p.m.
|
Breakout: Shifts Challenge: Robustness and Uncertainty under Real-World Distributional Shift
(
Breakout session
)
Schedule (GMT Timezone)
|
🔗 |
Tue 10:45 a.m. - 11:05 a.m.
|
VisDA21: Visual Domain Adaptation + Q&A
(
Talk
)
link »
SlidesLive Video » Progress in machine learning is typically measured by training and testing a model on the same distribution of data, i.e., the same domain. However, in real world applications, models often encounter out-of-distribution data. The VisDA21 competition invites methods that can adapt to novel test distributions and handle distributional shifts. Our task is object classification, but we measure accuracy on novel domains, rather than the traditional in-domain benchmarking. Teams will be given labeled source data and unlabeled target data from a different distribution (such as novel viewpoints, backgrounds, image quality). In addition, the target data may have missing and/or novel classes. Successful approaches will improve classification accuracy of known categories on target-domain data while learning to deal with missing and/or unknown categories. |
Kate Saenko · Kuniaki Saito · Donghyun Kim · Samarth Mishra · Ben Usman · Piotr Teterwak · Dina Bashkirova · Dan Hendrycks 🔗 |
Tue 11:04 a.m. - 2:04 p.m.
|
Breakout: VisDA21: Visual Domain Adaptation
(
Break out session
)
|
🔗 |
Tue 11:05 a.m. - 11:25 a.m.
|
HEAR 2021: Holistic Evaluation of Audio Representations + Q&A
(
Talk
)
link »
SlidesLive Video » Humans can infer a wide range of properties from a perceived sound, such as information about the source (e.g. what generated the sound? where is it coming from?), the information the sound conveys (this is a word that means X, this is a musical note in scale Y), and how it compares to other sounds (these two sounds come/don't come from the same source and are/aren't identical). Can any one learned representation do the same? The aim of this competition is to develop a general-purpose audio representation that provides a meaningful basis for learning in a wide variety of tasks and scenarios. We challenge participants with the following questions: Is it possible to develop a single representation that models all psychoacoustic phenomena? What approach best generalizes to a wide range of downstream audio tasks without fine-tuning? What audio representation allows researchers to formulate and solve novel and societally-valuable problems in simple, repeatable ways? We will evaluate audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, medical audio, and music. In the spirit of shared exchange, all participants must submit an audio embedding model, following a common API, that is general-purpose, open-source, and freely available to use. |
Joseph Turian · Jordan Shier · Bhiksha Raj · Bjoern Schuller · Christian Steinmetz · George Tzanetakis · Gissel Velarde · Kirk McNally · Max Henry · Nicolas Pinto · Yonatan Bisk · George Tzanetakis · Camille Noufi · Dorien Herremans · Jesse Engel · Justin Salamon · Prany Manocha · Philippe Esling · Shinji Watanabe
|
Tue 11:24 a.m. - 2:24 p.m.
|
Breakout: HEAR 2021: Holistic Evaluation of Audio Representations
(
Breakout session
)
Schedule (GMT Timezone) Invited speakers have been asked to prepare a 5-minute lightning talk. Following the lightning talks we will have an open question and discussion period. The breakout session will start at 19:25 GMT and last for approximately one hour. The current tentative team order for lightning talks is:
|
🔗 |
Tue 11:25 a.m. - 11:45 a.m.
|
WebQA Competition + Q&A
(
Talk
)
link »
SlidesLive Video » WebQA is a new benchmark for multimodal multihop reasoning in which systems are presented with the same style of data as humans when searching the web: snippets and images. Upon seeing a question, the system must identify which candidates potentially inform the answer from a candidate pool. Then the system is expected to aggregate information from selected candidates with reasoning to generate an answer in natural language form. Each datum is a question paired with a series of potentially long snippets or images that serve as "knowledge carriers" over which to reason. Systems will be evaluated on both supporting fact retrieval and answer generation to measure correctness and interpretability. To demonstrate multihop multimodal reasoning ability, models should be able to 1) understand and represent knowledge from different modalities, 2) identify and aggregate relevant knowledge fragments scattered across multiple sources, 3) make inference and do natural language generation. |
Yingshan CHANG · Yonatan Bisk · Mridu Narang · Levi Melnick · Jianfeng Gao · Hisami Suzuki · Guihong Cao 🔗 |
Tue 11:44 a.m. - 2:44 p.m.
|
Breakout: WebQA Competition
(
Breakout session
)
|
🔗 |
Tue 11:45 a.m. - 12:05 p.m.
|
Diamond: A MineRL Competition on Training Sample-Efficient Agents + Q&A
(
Talk
)
link »
SlidesLive Video » In the third MineRL Diamond competition, participants continue to develop algorithms which can efficiently leverage human demonstrations to drastically reduce the number of samples needed to solve a complex task in Minecraft. The competition environment features sparse-rewards, long-term planning, vision and sub-task hierarchies. To ensure that truly sample-efficient are developed, organizers re-train submitted systems on a fixed cloud-computing environment for a limited number of samples (4 days or 8 million samples). To ease the entry to machine learning research, the competition features two tracks: introduction, which allows agents developed using any method ranging from end-to-end machine learning solutions to programmatic approaches; and research, which requires participants develop novel imitation and reinforcement learning algorithms to solve this difficult sample-limited task. |
William Guss · Alara Dirik · Byron Galbraith · Brandon Houghton · Anssi Kanervisto · Noboru Kuno · Stephanie Milani · Sharada Mohanty · Karolis Ramanauskas · Ruslan Salakhutdinov · Rohin Shah · Nicholay Topin · Steven Wang · Cody Wild
|
Tue 12:04 p.m. - 3:04 p.m.
|
Breakout: Diamond: A MineRL Competition on Training Sample-Efficient Agents
(
Breakout session
)
|
🔗 |