Understanding the fidelity of approximate inference has extraordinary value beyond the standard approach of measuring generalization on a particular task: if approximate inference is working correctly, then we can expect more reliable and accurate deployment across any number of real-world settings. In this regular competition, we invite the community to evaluate the fidelity of approximate Bayesian inference procedures in deep learning, using as a reference Hamiltonian Monte Carlo (HMC) samples obtained by parallelizing computations over hundreds of tensor processing unit (TPU) devices. We consider a variety of tasks, including image recognition, regression, covariate shift, and medical applications, such as diagnosing diabetic retinopathy. All data are publicly available, and we will release several baselines, including stochastic MCMC, variational methods, and deep ensembles.

The NetHack Challenge is based on the NetHack Learning Environment (NLE), where teams will compete to build the best agents to play the game of NetHack. NetHack is a ASCII-rendered single-player dungeon crawl game that is one of the oldest and most difficult computer games in history. NetHack is procedurally-generated, with hundreds of different entities and complex environment dynamics, presenting an extremely challenging environment for both current state-of-the-art RL agents and humans, while crucially being lightning-fast to simulate. We are excited that this competition offers machine learning students, researchers and NetHack-bot builders the opportunity to participate in a grand challenge in AI without prohibitive computational costs—and we are eagerly looking forward to the wide variety of submissions.

The Machine Learning for Combinatorial Optimization (ML4CO) competition aims at improving a state-of-the-art mathematical solver by replacing key heuristic components with machine learning models trained on historical data. To that end participants will compete on the three following challenges, each corresponding to a distinct control task arising in a branch-and-bound solver: producing good solutions (primal task), proving optimality via branching (dual task), and choosing the best solver parameters (configuration task). Each task is exposed through an OpenAI-gym Python API build on top of the open-source solver SCIP, using the Ecole library. Participants can compete in any subset of the proposed challenges. While we encourage solutions derived from the reinforcement learning paradigm, any algorithmic solution respecting the competition's API is accepted.

Schedule (GMT Timezone)

• 19:05 GMT - competition wrap-up (15 min)
• 19:20 GMT - presentation from the winners (3x10 min)
• 19:50 GMT - poster session on Gathertown (30 min)
• 20:20 GMT - break (15 min)
• 20:35 GMT - keynote (30 min) - Kate Smith-Miles
• 21:05 GMT - panel discussion (1 hour) - Andrea Lodi, Timo Berthold, Frank Hutter, Vinod Nair, Kevin Leyton-Brown

Traffic is said to follow `hidden rules' that can be transferred across domain shifts. Our competition sets out to explore this meta topic with two few-shot learning tasks: predictions across a temporal shift brought about by COVID-19 and across a spatio-temporal shift in hitherto unseen cities. We provide an unprecedented, large data set from $10^{12}$ real world GPS probes in $10$ cities binned in space and time into multi-channel movie frames, as well as static data on the basic road connections of the underlying road network. Thus participants can approach these transfer tasks using graph based approaches encoding knowledge about the road network or approaches from computer vision like U-nets, which were highly successful in our previous competitions. Any advance in these questions will have a large impact on smart city planning, on mobility in general and thus, ultimately, our way of living more sustainably.

The Benchmark for Agents that Solve Almost-Lifelike Tasks (BASALT) competition aims to promote research in the area of learning from human feedback in order to enable agents that can pursue tasks that do not have crisp, easily defined reward functions. We provide tasks consisting of a simple English language description alongside a Gym environment, without any associated reward function, but with expert demos. Participants will train agents for these tasks using their preferred methods. We expect typical solutions will use imitation learning, or learning from comparisons. Submitted agents will be evaluated based on how well they complete the tasks, as judged by humans given the same description of the tasks.

Human intelligence has the remarkable ability to quickly adapt to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research in this direction, we propose IGLU: Interactive Grounded Language Understanding in a Collaborative Environment. The primary goal of the competition is to approach the problem of how to build interactive agents that learn to solve a task while provided with grounded natural language instructions in a collaborative environment. Understanding the complexity of the challenge, we split it into sub-tasks to make it feasible for participants.