Timezone: »
Bayesian optimization is a prominent method for optimizing expensive to evaluate black-box functions that is prominently applied to tuning the hyperparameters of machine learning algorithms. Despite its successes, the prototypical Bayesian optimization approach - using Gaussian process models - does not scale well to either many hyperparameters or many function evaluations. Attacking this lack of scalability and flexibility is thus one of the key challenges of the field. We present a general approach for using flexible parametric models (neural networks) for Bayesian optimization, staying as close to a truly Bayesian treatment as possible. We obtain scalability through stochastic gradient Hamiltonian Monte Carlo, whose robustness we improve via a scale adaptation. Experiments including multi-task Bayesian optimization with 21 tasks, parallel optimization of deep neural networks and deep reinforcement learning show the power and flexibility of this approach.
Author Information
Jost Tobias Springenberg (University of Freiburg)
Aaron Klein (University of Freiburg)
Stefan Falkner (University of Freiburg)
Frank Hutter (University of Freiburg)
Frank Hutter is a Full Professor for Machine Learning at the Computer Science Department of the University of Freiburg (Germany), where he previously was an assistant professor 2013-2017. Before that, he was at the University of British Columbia (UBC) for eight years, for his PhD and postdoc. Frank's main research interests lie in machine learning, artificial intelligence and automated algorithm design. For his 2009 PhD thesis on algorithm configuration, he received the CAIAC doctoral dissertation award for the best thesis in AI in Canada that year, and with his coauthors, he received several best paper awards and prizes in international competitions on machine learning, SAT solving, and AI planning. Since 2016 he holds an ERC Starting Grant for a project on automating deep learning based on Bayesian optimization, Bayesian neural networks, and deep reinforcement learning.
More from the Same Authors
-
2021 : OpenML Benchmarking Suites »
Bernd Bischl · Giuseppe Casalicchio · Matthias Feurer · Pieter Gijsbers · Frank Hutter · Michel Lang · Rafael Gomes Mantovani · Jan van Rijn · Joaquin Vanschoren -
2021 : HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO »
Katharina Eggensperger · Philipp Müller · Neeratyoy Mallik · Matthias Feurer · Rene Sass · Aaron Klein · Noor Awad · Marius Lindauer · Frank Hutter -
2021 : Transformers Can Do Bayesian-Inference By Meta-Learning on Prior-Data »
Samuel Müller · Noah Hollmann · Sebastian Pineda Arango · Josif Grabocka · Frank Hutter -
2022 : c-TPE: Generalizing Tree-structured Parzen Estimator with Inequality Constraints for Continuous and Categorical Hyperparameter Optimization »
Shuhei Watanabe · Frank Hutter -
2022 : TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second »
Noah Hollmann · Samuel Müller · Katharina Eggensperger · Frank Hutter -
2022 : On the Importance of Architectures and Hyperparameters for Fairness in Face Recognition »
Samuel Dooley · Rhea Sukthanker · John Dickerson · Colin White · Frank Hutter · Micah Goldblum -
2022 : Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks »
Steven Adriaensen · Herilalaina Rakotoarison · Samuel Müller · Frank Hutter -
2022 : Transfer NAS with Meta-learned Bayesian Surrogates »
Gresa Shala · Thomas Elsken · Frank Hutter · Josif Grabocka -
2022 : Gray-Box Gaussian Processes for Automated Reinforcement Learning »
Gresa Shala · André Biedenkapp · Frank Hutter · Josif Grabocka -
2022 : AutoRL-Bench 1.0 »
Gresa Shala · Sebastian Pineda Arango · André Biedenkapp · Frank Hutter · Josif Grabocka -
2022 : Bayesian Optimization with a Neural Network Meta-learned on Synthetic Data Only »
Samuel Müller · Sebastian Pineda Arango · Matthias Feurer · Josif Grabocka · Frank Hutter -
2022 : GraViT-E: Gradient-based Vision Transformer Search with Entangled Weights »
Rhea Sukthanker · Arjun Krishnakumar · sharat patil · Frank Hutter -
2022 : PriorBand: HyperBand + Human Expert Knowledge »
Neeratyoy Mallik · Carl Hvarfner · Danny Stoll · Maciej Janowski · Edward Bergman · Marius Lindauer · Luigi Nardi · Frank Hutter -
2022 : Towards Discovering Neural Architectures from Scratch »
Simon Schrodi · Danny Stoll · Robin Ru · Rhea Sukthanker · Thomas Brox · Frank Hutter -
2022 : On the Importance of Architectures and Hyperparameters for Fairness in Face Recognition »
Samuel Dooley · Rhea Sukthanker · John Dickerson · Colin White · Frank Hutter · Micah Goldblum -
2022 : Multi-objective Tree-structured Parzen Estimator Meets Meta-learning »
Shuhei Watanabe · Noor Awad · Masaki Onishi · Frank Hutter -
2022 Spotlight: Lightning Talks 2B-4 »
Feiyi Xiao · Amrutha Saseendran · Kwangho Kim · Keyu Yan · Changjian Shui · Guangxi Li · Shikun Li · Edward Kennedy · Man Zhou · Gezheng Xu · Ruilin Ye · Xiaobo Xia · Junjie Tang · Kathrin Skubch · Stefan Falkner · Hansong Zhang · Jose Zubizarreta · Huaying Fang · Xuanqiang Zhao · Jie Huang · Qi CHEN · Yibing Zhan · Jiaqi Li · Xin Wang · Ruibin Xi · Feng Zhao · Margret Keuper · Charles Ling · Shiming Ge · Chengjun Xie · Tongliang Liu · Tal Arbel · Chongyi Li · Danfeng Hong · Boyu Wang · Christian Gagné -
2022 Spotlight: Trading off Image Quality for Robustness is not Necessary with Regularized Deterministic Autoencoders »
Amrutha Saseendran · Kathrin Skubch · Stefan Falkner · Margret Keuper -
2022 : TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second »
Noah Hollmann · Samuel Müller · Katharina Eggensperger · Frank Hutter -
2022 Poster: Joint Entropy Search For Maximally-Informed Bayesian Optimization »
Carl Hvarfner · Frank Hutter · Luigi Nardi -
2022 Poster: Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design »
Jörg Franke · Frederic Runge · Frank Hutter -
2022 Poster: NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies »
Arjun Krishnakumar · Colin White · Arber Zela · Renbo Tu · Mahmoud Safari · Frank Hutter -
2022 Poster: JAHS-Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search »
Archit Bansal · Danny Stoll · Maciej Janowski · Arber Zela · Frank Hutter -
2021 : CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning »
Carolin Benjamins · Theresa Eimer · Frederik Schubert · André Biedenkapp · Bodo Rosenhahn · Frank Hutter · Marius Lindauer -
2021 Workshop: 5th Workshop on Meta-Learning »
Erin Grant · Fábio Ferreira · Frank Hutter · Jonathan Richard Schwarz · Joaquin Vanschoren · Huaxiu Yao -
2021 Poster: How Powerful are Performance Predictors in Neural Architecture Search? »
Colin White · Arber Zela · Robin Ru · Yang Liu · Frank Hutter -
2021 Poster: Well-tuned Simple Nets Excel on Tabular Datasets »
Arlind Kadra · Marius Lindauer · Frank Hutter · Josif Grabocka -
2021 Poster: NAS-Bench-x11 and the Power of Learning Curves »
Shen Yan · Colin White · Yash Savani · Frank Hutter -
2021 Poster: Shape your Space: A Gaussian Mixture Regularization Approach to Deterministic Autoencoders »
Amrutha Saseendran · Kathrin Skubch · Stefan Falkner · Margret Keuper -
2021 Poster: Neural Ensemble Search for Uncertainty Estimation and Dataset Shift »
Sheheryar Zaidi · Arber Zela · Thomas Elsken · Chris C Holmes · Frank Hutter · Yee Teh -
2020 : Q/A for invited talk #1 »
Frank Hutter -
2020 : Meta-learning neural architectures, initial weights, hyperparameters, and algorithm components »
Frank Hutter -
2019 : Frank Hutter (University of Freiburg) "A Proposal for a New Competition Design Emphasizing Scientific Insights" »
Frank Hutter -
2019 Workshop: Meta-Learning »
Roberto Calandra · Ignasi Clavera Gilaberte · Frank Hutter · Joaquin Vanschoren · Jane Wang -
2019 Poster: Meta-Surrogate Benchmarking for Hyperparameter Optimization »
Aaron Klein · Zhenwen Dai · Frank Hutter · Neil Lawrence · Javier González -
2018 Workshop: NIPS 2018 Workshop on Meta-Learning »
Joaquin Vanschoren · Frank Hutter · Sachin Ravi · Jane Wang · Erin Grant -
2018 Poster: Maximizing acquisition functions for Bayesian optimization »
James Wilson · Frank Hutter · Marc Deisenroth -
2018 Tutorial: Automatic Machine Learning »
Frank Hutter · Joaquin Vanschoren -
2017 Workshop: Workshop on Meta-Learning »
Roberto Calandra · Frank Hutter · Hugo Larochelle · Sergey Levine -
2016 : Invited talk, Frank Hutter »
Frank Hutter -
2016 Workshop: Bayesian Optimization: Black-box Optimization and Beyond »
Roberto Calandra · Bobak Shahriari · Javier Gonzalez · Frank Hutter · Ryan Adams -
2016 : Frank Hutter (University Freiburg) »
Frank Hutter -
2016 Oral: Bayesian Optimization with Robust Bayesian Neural Networks »
Jost Tobias Springenberg · Aaron Klein · Stefan Falkner · Frank Hutter -
2015 : Scalable and Flexible Bayesian Optimization for Algorithm Configuration »
Frank Hutter -
2015 Poster: Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images »
Manuel Watter · Jost Springenberg · Joschka Boedecker · Martin Riedmiller -
2015 Poster: Efficient and Robust Automated Machine Learning »
Matthias Feurer · Aaron Klein · Katharina Eggensperger · Jost Springenberg · Manuel Blum · Frank Hutter -
2014 Poster: Discriminative Unsupervised Feature Learning with Convolutional Neural Networks »
Alexey Dosovitskiy · Jost Tobias Springenberg · Martin Riedmiller · Thomas Brox