Timezone: »
A Bayesian treatment can mitigate overconfidence in ReLU nets around the training data. But far away from them, ReLU Bayesian neural networks (BNNs) can still underestimate uncertainty and thus be asymptotically overconfident. This issue arises since the output variance of a BNN with finitely many features is quadratic in the distance from the data region. Meanwhile, Bayesian linear models with ReLU features converge, in the infinite-width limit, to a particular Gaussian process (GP) with a variance that grows cubically so that no asymptotic overconfidence can occur. While this may seem of mostly theoretical interest, in this work, we show that it can be used in practice to the benefit of BNNs. We extend finite ReLU BNNs with infinite ReLU features via the GP and show that the resulting model is asymptotically maximally uncertain far away from the data while the BNNs' predictive power is unaffected near the data. Although the resulting model approximates a full GP posterior, thanks to its structure, it can be applied post-hoc to any pre-trained ReLU BNN at a low cost.
Author Information
Agustinus Kristiadi (University of Tübingen)
Matthias Hein (University of Tübingen)
Philipp Hennig (University of Tübingen and MPI Tübingen)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Poster: An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence »
Tue. Dec 7th 04:30 -- 06:00 PM Room Virtual
More from the Same Authors
-
2021 : RobustBench: a standardized adversarial robustness benchmark »
Francesco Croce · Maksym Andriushchenko · Vikash Sehwag · Edoardo Debenedetti · Nicolas Flammarion · Mung Chiang · Prateek Mittal · Matthias Hein -
2021 : Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning »
Runa Eschenhagen · Erik Daxberger · Philipp Hennig · Agustinus Kristiadi -
2021 : Being a Bit Frequentist Improves Bayesian Neural Networks »
Agustinus Kristiadi · Matthias Hein · Philipp Hennig -
2021 Poster: Laplace Redux - Effortless Bayesian Deep Learning »
Erik Daxberger · Agustinus Kristiadi · Alexander Immer · Runa Eschenhagen · Matthias Bauer · Philipp Hennig -
2021 Poster: A Probabilistic State Space Model for Joint Inference from Differential Equations and Data »
Jonathan Schmidt · Nicholas Krämer · Philipp Hennig -
2021 Poster: Linear-Time Probabilistic Solution of Boundary Value Problems »
Nicholas Krämer · Philipp Hennig -
2021 Poster: Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks »
Frank Schneider · Felix Dangel · Philipp Hennig -
2021 Poster: Meta-Learning the Search Distribution of Black-Box Random Search Based Adversarial Attacks »
Maksym Yatsura · Jan Metzen · Matthias Hein -
2017 : Poster Spotlights I »
Taesik Na · Yang Song · Aman Sinha · Richard Shin · Qiuyuan Huang · Nina Narodytska · Matt Staib · Kexin Pei · Fnu Suya · Amirata Ghorbani · Jacob Buckman · Matthias Hein · Huan Zhang · Yanjun Qi · Yuan Tian · Min Du · Dimitris Tsipras -
2017 Poster: Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation »
Matthias Hein · Maksym Andriushchenko -
2016 Workshop: Optimizing the Optimizers »
Maren Mahsereci · Alex Davies · Philipp Hennig -
2016 Poster: Clustering Signed Networks with the Geometric Mean of Laplacians »
Pedro Mercado · Francesco Tudisco · Matthias Hein -
2016 Poster: Globally Optimal Training of Generalized Polynomial Neural Networks with Nonlinear Spectral Methods »
Antoine Gautier · Quynh Nguyen · Matthias Hein -
2015 Workshop: Probabilistic Integration »
Michael A Osborne · Philipp Hennig -
2015 Poster: Efficient Output Kernel Learning for Multiple Tasks »
Pratik Kumar Jawanpuria · Maksim Lapin · Matthias Hein · Bernt Schiele -
2015 Poster: Probabilistic Line Searches for Stochastic Optimization »
Maren Mahsereci · Philipp Hennig -
2015 Poster: Top-k Multiclass SVM »
Maksim Lapin · Matthias Hein · Bernt Schiele -
2015 Oral: Probabilistic Line Searches for Stochastic Optimization »
Maren Mahsereci · Philipp Hennig -
2015 Spotlight: Top-k Multiclass SVM »
Maksim Lapin · Matthias Hein · Bernt Schiele -
2015 Poster: Regularization-Free Estimation in Trace Regression with Symmetric Positive Semidefinite Matrices »
Martin Slawski · Ping Li · Matthias Hein -
2014 Poster: Incremental Local Gaussian Regression »
Franziska Meier · Philipp Hennig · Stefan Schaal -
2014 Poster: Tight Continuous Relaxation of the Balanced k-Cut Problem »
Syama Sundar Rangapuram · Pramod Kaushik Mudrakarta · Matthias Hein -
2014 Poster: Probabilistic ODE Solvers with Runge-Kutta Means »
Michael Schober · David Duvenaud · Philipp Hennig -
2014 Poster: Sampling for Inference in Probabilistic Models with Fast Bayesian Quadrature »
Tom Gunter · Michael A Osborne · Roman Garnett · Philipp Hennig · Stephen J Roberts -
2014 Oral: Probabilistic ODE Solvers with Runge-Kutta Means »
Michael Schober · David Duvenaud · Philipp Hennig -
2013 Workshop: Bayesian Optimization in Theory and Practice »
Matthew Hoffman · Jasper Snoek · Nando de Freitas · Michael A Osborne · Ryan Adams · Sebastien Bubeck · Philipp Hennig · Remi Munos · Andreas Krause -
2013 Poster: The Randomized Dependence Coefficient »
David Lopez-Paz · Philipp Hennig · Bernhard Schölkopf -
2013 Poster: The Total Variation on Hypergraphs - Learning on Hypergraphs Revisited »
Matthias Hein · Simon Setzer · Leonardo Jost · Syama Sundar Rangapuram -
2013 Spotlight: The Total Variation on Hypergraphs - Learning on Hypergraphs Revisited »
Matthias Hein · Simon Setzer · Leonardo Jost · Syama Sundar Rangapuram -
2013 Poster: Matrix factorization with binary components »
Martin Slawski · Matthias Hein · Pavlo Lutsik -
2013 Spotlight: Matrix factorization with binary components »
Martin Slawski · Matthias Hein · Pavlo Lutsik -
2012 Workshop: Probabilistic Numerics »
Philipp Hennig · John P Cunningham · Michael A Osborne -
2011 Poster: Sparse recovery by thresholded non-negative least squares »
Martin Slawski · Matthias Hein -
2011 Poster: Beyond Spectral Clustering - Tight Relaxations of Balanced Graph Cuts »
Matthias Hein · Simon Setzer -
2011 Poster: Optimal Reinforcement Learning for Gaussian Systems »
Philipp Hennig -
2010 Poster: An Inverse Power Method for Nonlinear Eigenproblems with Applications in 1-Spectral Clustering and Sparse PCA »
Matthias Hein · Thomas Bühler -
2010 Spotlight: Getting lost in space: Large sample analysis of the resistance distance »
Ulrike von Luxburg · Agnes Radl · Matthias Hein -
2010 Poster: Getting lost in space: Large sample analysis of the resistance distance »
Ulrike von Luxburg · Agnes Radl · Matthias Hein -
2009 Poster: Semi-supervised Regression using Hessian energy with an application to semi-supervised dimensionality reduction »
Kwang In Kim · Florian Steinke · Matthias Hein -
2009 Poster: Robust Nonparametric Regression with Metric-Space Valued Output »
Matthias Hein -
2008 Poster: Non-parametric Regression Between Manifolds »
Florian Steinke · Matthias Hein -
2008 Poster: Influence of graph construction on graph-based clustering measures »
Markus M Maier · Ulrike von Luxburg · Matthias Hein -
2008 Oral: Influence of graph construction on graph-based clustering measures »
Markus M Maier · Ulrike von Luxburg · Matthias Hein -
2006 Poster: Manifold Denoising »
Matthias Hein · Markus M Maier -
2006 Talk: Manifold Denoising »
Matthias Hein · Markus M Maier