BO-Arena: An Evolving Benchmark for High-Dimensional Bayesian Optimisation

High-dimensional Bayesian optimisation (HDBO) is an increasingly crowded field, with new methods proposed at every major conference, making it ever harder to answer a deceptively simple question: "Which method is currently state-of-the-art?" In this paper, we question the efficacy of benchmarking practices in the field, and find them to be systematically ineffective. Specifically, multiple published works fail to benchmark against the strongest baselines available at the time of writing, or even include incorrect implementations of key methods. As a remedy, we introduce BO-Arena, an actively maintained and extensible package providing canonical implementations of state-of-the-art algorithms, designed to streamline building new methods and rigorously benchmark against existing approaches. We use the package to ask what currently drives performance in HDBO, and find that the answer is even simpler than previously thought. Following these simple principles, we propose a new algorithm which achieves state-of-the-art performance on problems with large numbers of observations, outperforming considerably more complex recent methods. Finally, we question whether there is currently sufficient evidence to suggest that a simple Gaussian process baseline is outperformed by any more complex methods for HDBO.