Timezone: »
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create and leverage these benchmarking suites. These are seamlessly integrated into the OpenML platform, and accessible through interfaces in Python, Java, and R. OpenML benchmarking suites (a) are easy to use through standardized data formats, APIs, and client libraries; (b) come with extensive meta-information on the included datasets; and (c) allow benchmarks to be shared and reused in future studies. We then present a first, carefully curated and practical benchmarking suite for classification: the OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18). Finally, we discuss use cases and applications which demonstrate the usefulness of OpenML benchmarking suites and the OpenML-CC18 in particular.
Author Information
Bernd Bischl (LMU Munich)
Giuseppe Casalicchio (LMU Munich)
Matthias Feurer (University of Freiburg)
Pieter Gijsbers (Eindhoven University of Technology)
Frank Hutter (University of Freiburg & Bosch)
Frank Hutter is a Full Professor for Machine Learning at the Computer Science Department of the University of Freiburg (Germany), where he previously was an assistant professor 2013-2017. Before that, he was at the University of British Columbia (UBC) for eight years, for his PhD and postdoc. Frank's main research interests lie in machine learning, artificial intelligence and automated algorithm design. For his 2009 PhD thesis on algorithm configuration, he received the CAIAC doctoral dissertation award for the best thesis in AI in Canada that year, and with his coauthors, he received several best paper awards and prizes in international competitions on machine learning, SAT solving, and AI planning. Since 2016 he holds an ERC Starting Grant for a project on automating deep learning based on Bayesian optimization, Bayesian neural networks, and deep reinforcement learning.
Michel Lang
Rafael Gomes Mantovani (Federal Technology University of Paraná)
Jan van Rijn (Columbia University)
Joaquin Vanschoren (Eindhoven University of Technology)

Joaquin Vanschoren is Associate Professor in Machine Learning at the Eindhoven University of Technology. He holds a PhD from the Katholieke Universiteit Leuven, Belgium. His research focuses on understanding and automating machine learning, meta-learning, and continual learning. He founded and leads OpenML.org, a popular open science platform with over 250,000 users that facilitates the sharing and reuse of machine learning datasets and models. He is a founding member of the European AI networks ELLIS and CLAIRE, and an active member of MLCommons. He obtained several awards, including an Amazon Research Award, an ECMLPKDD Best Demo award, and the Dutch Data Prize. He was a tutorial speaker at NeurIPS 2018 and AAAI 2021, and gave over 30 invited talks. He co-initiated the NeurIPS Datasets and Benchmarks track and was NeurIPS Datasets and Benchmarks Chair from 2021 to 2023. He also co-organized the AutoML workshop series at ICML, and the Meta-Learning workshop series at NeurIPS. He is editor-in-chief of DMLR (part of JMLR), as well as an action editor for JMLR and machine learning moderator for ArXiv. He authored and co-authored over 150 scientific papers, as well as reference books on Automated Machine Learning and Meta-learning.
More from the Same Authors
-
2021 : HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO »
Katharina Eggensperger · Philipp Müller · Neeratyoy Mallik · Matthias Feurer · Rene Sass · Aaron Klein · Noor Awad · Marius Lindauer · Frank Hutter -
2021 : Towards modelling hazard factors in unstructured data spaces using gradient-based latent interpolation »
Tobias Weber · Michael Ingrisch · Bernd Bischl · David Rügamer -
2021 : Variational Task Encoders for Model-Agnostic Meta-Learning »
Joaquin Vanschoren -
2021 : Open-Ended Learning Strategies for Learning Complex Locomotion Skills »
Joaquin Vanschoren -
2021 : Transformers Can Do Bayesian-Inference By Meta-Learning on Prior-Data »
Samuel Müller · Noah Hollmann · Sebastian Pineda Arango · Josif Grabocka · Frank Hutter -
2021 : A Preliminary Study on the Feature Representations of Transfer Learning and Gradient-Based Meta-Learning Techniques »
Mike Huisman · Jan van Rijn · Aske Plaat -
2021 : Towards modelling hazard factors in unstructured data spaces using gradient-based latent interpolation »
Tobias Weber · Michael Ingrisch · Bernd Bischl · David Rügamer -
2022 : c-TPE: Generalizing Tree-structured Parzen Estimator with Inequality Constraints for Continuous and Categorical Hyperparameter Optimization »
Shuhei Watanabe · Frank Hutter -
2022 : TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second »
Noah Hollmann · Samuel Müller · Katharina Eggensperger · Frank Hutter -
2022 : On the Importance of Architectures and Hyperparameters for Fairness in Face Recognition »
Samuel Dooley · Rhea Sukthanker · John Dickerson · Colin White · Frank Hutter · Micah Goldblum -
2022 : Fifteen-minute Competition Overview Video »
Dustin Carrión-Ojeda · Ihsan Ullah · Sergio Escalera · Isabelle Guyon · Felix Mohr · Manh Hung Nguyen · Joaquin Vanschoren -
2022 : LOTUS: Learning to learn with Optimal Transport in Unsupervised Scenarios »
prabhant singh · Joaquin Vanschoren -
2022 : Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks »
Steven Adriaensen · Herilalaina Rakotoarison · Samuel Müller · Frank Hutter -
2022 : Transfer NAS with Meta-learned Bayesian Surrogates »
Gresa Shala · Thomas Elsken · Frank Hutter · Josif Grabocka -
2022 : Gray-Box Gaussian Processes for Automated Reinforcement Learning »
Gresa Shala · André Biedenkapp · Frank Hutter · Josif Grabocka -
2022 : AutoRL-Bench 1.0 »
Gresa Shala · Sebastian Pineda Arango · André Biedenkapp · Frank Hutter · Josif Grabocka -
2022 : Bayesian Optimization with a Neural Network Meta-learned on Synthetic Data Only »
Samuel Müller · Sebastian Pineda Arango · Matthias Feurer · Josif Grabocka · Frank Hutter -
2022 : GraViT-E: Gradient-based Vision Transformer Search with Entangled Weights »
Rhea Sukthanker · Arjun Krishnakumar · sharat patil · Frank Hutter -
2022 : PriorBand: HyperBand + Human Expert Knowledge »
Neeratyoy Mallik · Carl Hvarfner · Danny Stoll · Maciej Janowski · Edward Bergman · Marius Lindauer · Luigi Nardi · Frank Hutter -
2022 : Towards Discovering Neural Architectures from Scratch »
Simon Schrodi · Danny Stoll · Robin Ru · Rhea Sukthanker · Thomas Brox · Frank Hutter -
2022 : On the Importance of Architectures and Hyperparameters for Fairness in Face Recognition »
Samuel Dooley · Rhea Sukthanker · John Dickerson · Colin White · Frank Hutter · Micah Goldblum -
2022 : Multi-objective Tree-structured Parzen Estimator Meets Meta-learning »
Shuhei Watanabe · Noor Awad · Masaki Onishi · Frank Hutter -
2023 Poster: Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks »
Steven Adriaensen · Herilalaina Rakotoarison · Samuel Müller · Frank Hutter -
2023 Poster: PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning »
Neeratyoy Mallik · Carl Hvarfner · Edward Bergman · Danny Stoll · Maciej Janowski · Marius Lindauer · Luigi Nardi · Frank Hutter -
2023 Poster: Self-Correcting Bayesian Optimization through Bayesian Active Learning »
Carl Hvarfner · Erik Hellsten · Frank Hutter · Luigi Nardi -
2023 Poster: Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars »
Simon Schrodi · Danny Stoll · Binxin Ru · Rhea Sukthanker · Thomas Brox · Frank Hutter -
2023 Poster: LLMs for Semi-Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering »
Noah Hollmann · Samuel Müller · Frank Hutter -
2023 Poster: Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition »
Samuel Dooley · Rhea Sukthanker · John Dickerson · Colin White · Frank Hutter · Micah Goldblum -
2023 Poster: DataPerf: Benchmarks for Data-Centric AI Development »
Mark Mazumder · Colby Banbury · Xiaozhe Yao · Bojan Karlaš · William Gaviria Rojas · Sudnya Diamos · Greg Diamos · Lynn He · Alicia Parrish · Hannah Rose Kirk · Jessica Quaye · Charvi Rastogi · Douwe Kiela · David Jurado · David Kanter · Rafael Mosquera · Will Cukierski · Juan Ciro · Lora Aroyo · Bilge Acun · Lingjiao Chen · Mehul Raje · Max Bartolo · Evan Sabri Eyuboglu · Amirata Ghorbani · Emmett Goodman · Addison Howard · Oana Inel · Tariq Kane · Christine R. Kirkpatrick · D. Sculley · Tzu-Sheng Kuo · Jonas Mueller · Tristan Thrush · Joaquin Vanschoren · Margaret Warren · Adina Williams · Serena Yeung · Newsha Ardalani · Praveen Paritosh · Ce Zhang · James Zou · Carole-Jean Wu · Cody Coleman · Andrew Ng · Peter Mattson · Vijay Janapa Reddi -
2023 Oral: Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition »
Samuel Dooley · Rhea Sukthanker · John Dickerson · Colin White · Frank Hutter · Micah Goldblum -
2022 : AutoML for Neural Network Robustness Verification »
Jan van Rijn -
2022 : Towards better benchmarks for AutoML, meta-learning and continual learning in computer vision »
Joaquin Vanschoren -
2022 Competition: Cross-Domain MetaDL: Any-Way Any-Shot Learning Competition with Novel Datasets from Practical Domains »
Dustin Carrión-Ojeda · Ihsan Ullah · Sergio Escalera · Isabelle Guyon · Felix Mohr · Manh Hung Nguyen · Joaquin Vanschoren -
2022 : TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second »
Noah Hollmann · Samuel Müller · Katharina Eggensperger · Frank Hutter -
2022 Workshop: NeurIPS 2022 Workshop on Meta-Learning »
Huaxiu Yao · Eleni Triantafillou · Fabio Ferreira · Joaquin Vanschoren · Qi Lei -
2022 Poster: Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification »
Ihsan Ullah · Dustin Carrión-Ojeda · Sergio Escalera · Isabelle Guyon · Mike Huisman · Felix Mohr · Jan N. van Rijn · Haozhe Sun · Joaquin Vanschoren · Phan Anh Vu -
2022 Poster: Joint Entropy Search For Maximally-Informed Bayesian Optimization »
Carl Hvarfner · Frank Hutter · Luigi Nardi -
2022 Poster: Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design »
Jörg Franke · Frederic Runge · Frank Hutter -
2022 Poster: NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies »
Arjun Krishnakumar · Colin White · Arber Zela · Renbo Tu · Mahmoud Safari · Frank Hutter -
2022 Poster: JAHS-Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search »
Archit Bansal · Danny Stoll · Maciej Janowski · Arber Zela · Frank Hutter -
2021 : CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning »
Carolin Benjamins · Theresa Eimer · Frederik Schubert · André Biedenkapp · Bodo Rosenhahn · Frank Hutter · Marius Lindauer -
2021 Workshop: Data Centric AI »
Andrew Ng · Lora Aroyo · Greg Diamos · Cody Coleman · Vijay Janapa Reddi · Joaquin Vanschoren · Carole-Jean Wu · Sharon Zhou · Lynn He -
2021 Workshop: 5th Workshop on Meta-Learning »
Erin Grant · Fábio Ferreira · Frank Hutter · Jonathan Richard Schwarz · Joaquin Vanschoren · Huaxiu Yao -
2021 Poster: How Powerful are Performance Predictors in Neural Architecture Search? »
Colin White · Arber Zela · Robin Ru · Yang Liu · Frank Hutter -
2021 Datasets and Benchmarks: Dataset and Benchmark Poster Session 4 »
Joaquin Vanschoren · Serena Yeung -
2021 Datasets and Benchmarks: Dataset and Benchmark Track 3 »
Joaquin Vanschoren · Serena Yeung -
2021 Datasets and Benchmarks: Dataset and Benchmark Symposium »
Joaquin Vanschoren · Serena Yeung -
2021 Datasets and Benchmarks: Dataset and Benchmark Poster Session 3 »
Joaquin Vanschoren · Serena Yeung -
2021 Poster: Well-tuned Simple Nets Excel on Tabular Datasets »
Arlind Kadra · Marius Lindauer · Frank Hutter · Josif Grabocka -
2021 Poster: NAS-Bench-x11 and the Power of Learning Curves »
Shen Yan · Colin White · Yash Savani · Frank Hutter -
2021 Datasets and Benchmarks: Dataset and Benchmark Track 2 »
Joaquin Vanschoren · Serena Yeung -
2021 Panel: The Role of Benchmarks in the Scientific Progress of Machine Learning »
Lora Aroyo · Samuel Bowman · Isabelle Guyon · Joaquin Vanschoren -
2021 : MetaDL: Few Shot Learning Competition with Novel Datasets from Practical Domains + Q&A »
Adrian El Baz · Isabelle Guyon · Zhengying Liu · Jan N. Van Rijn · Haozhe Sun · Sébastien Treguer · Wei-Wei Tu · Ihsan Ullah · Joaquin Vanschoren · Phan Ahn Vu -
2021 Datasets and Benchmarks: Dataset and Benchmark Poster Session 2 »
Joaquin Vanschoren · Serena Yeung -
2021 Poster: Explaining Hyperparameter Optimization via Partial Dependence Plots »
Julia Moosbauer · Julia Herbinger · Giuseppe Casalicchio · Marius Lindauer · Bernd Bischl -
2021 Poster: Neural Ensemble Search for Uncertainty Estimation and Dataset Shift »
Sheheryar Zaidi · Arber Zela · Thomas Elsken · Chris C Holmes · Frank Hutter · Yee Teh -
2021 Datasets and Benchmarks: Dataset and Benchmark Poster Session 1 »
Joaquin Vanschoren · Serena Yeung -
2021 Datasets and Benchmarks: Dataset and Benchmark Track 1 »
Joaquin Vanschoren · Serena Yeung -
2020 : Introduction for invited speaker, Louis Kirsch »
Joaquin Vanschoren -
2020 : Q/A for invited talk #1 »
Frank Hutter -
2020 : Meta-learning neural architectures, initial weights, hyperparameters, and algorithm components »
Frank Hutter -
2020 Workshop: Meta-Learning »
Jane Wang · Joaquin Vanschoren · Erin Grant · Jonathan Richard Schwarz · Francesco Visin · Jeff Clune · Roberto Calandra -
2019 : Frank Hutter (University of Freiburg) "A Proposal for a New Competition Design Emphasizing Scientific Insights" »
Frank Hutter -
2019 Workshop: Meta-Learning »
Roberto Calandra · Ignasi Clavera Gilaberte · Frank Hutter · Joaquin Vanschoren · Jane Wang -
2019 Poster: Meta-Surrogate Benchmarking for Hyperparameter Optimization »
Aaron Klein · Zhenwen Dai · Frank Hutter · Neil Lawrence · Javier González -
2018 : Meta Learning for Defaults - Symbolic Defaults »
Jan van Rijn -
2018 Workshop: NIPS 2018 Workshop on Meta-Learning »
Joaquin Vanschoren · Frank Hutter · Sachin Ravi · Jane Wang · Erin Grant -
2018 Poster: Maximizing acquisition functions for Bayesian optimization »
James Wilson · Frank Hutter · Marc Deisenroth -
2018 Tutorial: Automatic Machine Learning »
Frank Hutter · Joaquin Vanschoren -
2017 Workshop: Workshop on Meta-Learning »
Roberto Calandra · Frank Hutter · Hugo Larochelle · Sergey Levine -
2016 : Invited talk, Frank Hutter »
Frank Hutter -
2016 Workshop: Bayesian Optimization: Black-box Optimization and Beyond »
Roberto Calandra · Bobak Shahriari · Javier Gonzalez · Frank Hutter · Ryan Adams -
2016 : Frank Hutter (University Freiburg) »
Frank Hutter -
2016 : OpenML in research and education »
Joaquin Vanschoren -
2016 Poster: Bayesian Optimization with Robust Bayesian Neural Networks »
Jost Tobias Springenberg · Aaron Klein · Stefan Falkner · Frank Hutter -
2016 Oral: Bayesian Optimization with Robust Bayesian Neural Networks »
Jost Tobias Springenberg · Aaron Klein · Stefan Falkner · Frank Hutter -
2015 : Scalable and Flexible Bayesian Optimization for Algorithm Configuration »
Frank Hutter -
2015 Poster: Efficient and Robust Automated Machine Learning »
Matthias Feurer · Aaron Klein · Katharina Eggensperger · Jost Springenberg · Manuel Blum · Frank Hutter