Timezone: »

 
Poster
Dangers of Bayesian Model Averaging under Covariate Shift
Pavel Izmailov · Patrick Nicholson · Sanae Lotfi · Andrew Wilson

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @
in Poster Session 1 »

Approximate Bayesian inference for neural networks is considered a robust alternative to standard training, often providing good performance on out-of-distribution data. However, Bayesian neural networks (BNNs) with high-fidelity approximate inference via full-batch Hamiltonian Monte Carlo achieve poor generalization under covariate shift, even underperforming classical estimation. We explain this surprising result, showing how a Bayesian model average can in fact be problematic under covariate shift, particularly in cases where linear dependencies in the input features cause a lack of posterior contraction. We additionally show why the same issue does not affect many approximate inference procedures, or classical maximum a-posteriori (MAP) training. Finally, we propose novel priors that improve the robustness of BNNs to many sources of covariate shift.

Keywords: Deep Learning · Robustness

Author Information

Pavel Izmailov (New York University)
Patrick Nicholson (Covera Health)
Sanae Lotfi (New York University)
Andrew Wilson (New York University)
Andrew Wilson

I am a professor of machine learning at New York University.

More from the Same Authors