In this work we use variational inference to quantify the degree of epistemic uncertainty in model predictions of radio galaxy classification and show that the level of model posterior variance for individual test samples is correlated with measures of human uncertainty when labelling radio galaxies. Using the posterior distributions for individual weights, we show that signal-to-noise ratio (SNR) ranking allows pruning of the fully-connected layers to the level of 40% without significant loss of performance, and that this pruning reduces the predictive uncertainty in the model. Finally we show that, like other work in this field, we experience a cold posterior effect. We examine whether the inclusion of an additional variance term in the loss can compensate for this effect, but find that it does not make a significant difference. We interpret this as the cold posterior effect being due to the overly effective curation of our training sample rather than model misspecification and raise this as a potential issue for Bayesian approaches to radio galaxy classification in future.