Machine learning research has benefited considerably from the adoption of standardised public benchmarks. While the importance of these benchmarks is undisputed, we argue against the current incentive system and its heavy reliance upon performance as a proxy for scientific progress. The status quo incentivises researchers to “beat the state of the art”, potentially at the expense of deep scientific understanding and rigorous experimental design. Since typically only positive results are rewarded, the negative results inevitably encountered during research are often omitted, allowing many other groups to unknowingly and wastefully repeat these negative findings.

Pre-registration is a publishing and reviewing model that aims to address these issues by changing the incentive system. A pre-registered paper is a regular paper that is submitted for peer-review without any experimental results, describing instead an experimental protocol to be followed after the paper is accepted. This implies that it is important for the authors to make compelling arguments from theory or past published evidence. As for reviewers, they must assess these arguments together with the quality of the experimental design, rather than comparing numeric results. While pre-registration has been highly adopted in fields such as medicine and psychology, there is little such experience inthe machine learning community. In this workshop, we propose to conduct a full pre-registration review-cycle for machine learning. Our proposal follows an initial small-scale trial of pre-registration in computer vision (Henriques et al., 2019) and builds on a successful pilot study in pre-registration at NeurIPS 2020 (Bertinetto et al., 2020). We have already received a number of requests to repeat the workshop, indicating strong community interest.