Vincent Fortuin---Bayesian Neural Network Priors Revisited

Isotropic Gaussian priors are the de facto standard for modern Bayesian neural network inference. However, there has been recent controversy over the question whether they might be to blame for the undesirable cold posterior effect. We study this question empirically and find that for densely connected networks, Gaussian priors are indeed less well suited than more heavy-tailed ones. Conversely, for convolutional architectures, Gaussian priors seem to perform well and thus cannot fully explain the cold posterior effect. These findings coincide with the empirical maximum-likelihood weight distributions discovered by standard gradient-based training.