Skip to yearly menu bar Skip to main content


Bayesian Optimization with Robust Bayesian Neural Networks

Jost Tobias Springenberg · Aaron Klein · Stefan Falkner · Frank Hutter

Area 5+6+7+8 #189

Keywords: [ MCMC ] [ Deep Learning or Neural Networks ]


Bayesian optimization is a prominent method for optimizing expensive to evaluate black-box functions that is prominently applied to tuning the hyperparameters of machine learning algorithms. Despite its successes, the prototypical Bayesian optimization approach - using Gaussian process models - does not scale well to either many hyperparameters or many function evaluations. Attacking this lack of scalability and flexibility is thus one of the key challenges of the field. We present a general approach for using flexible parametric models (neural networks) for Bayesian optimization, staying as close to a truly Bayesian treatment as possible. We obtain scalability through stochastic gradient Hamiltonian Monte Carlo, whose robustness we improve via a scale adaptation. Experiments including multi-task Bayesian optimization with 21 tasks, parallel optimization of deep neural networks and deep reinforcement learning show the power and flexibility of this approach.

Live content is unavailable. Log in and register to view live content