Timezone: »

 
Tutorial
(Track2) Practical Uncertainty Estimation and Out-of-Distribution Robustness in Deep Learning Q&A
Dustin Tran · Balaji Lakshminarayanan · Jasper Snoek

Wed Dec 09 12:00 PM -- 12:50 PM (PST) @

Deep learning models are bad at signalling failure: They tend to make predictions with high confidence, and this is problematic in real-world applications such as healthcare, self-driving cars, and natural language systems, where there are considerable safety implications, or where there are discrepancies between the training data and data that the model makes predictions on. There is a pressing need both for understanding when models should not make predictions and improving model robustness to natural changes in the data.

This tutorial will give an overview of the landscape of uncertainty and robustness in deep learning. Namely, we examine calibration and out-of-distribution generalization as key tasks. Then we will go into a deep dive into promising avenues. This includes methods which average over multiple neural network predictions such as Bayesian neural nets, ensembles, and Gaussian processes; methods on the frontier of scale in terms of their overall parameter or prediction-time efficiency; and methods which encourage key inductive biases such as data augmentation. We ground these ideas in both empirical understanding and theory, and we provide practical recommendations with baselines and tips & tricks. Finally, we highlight open challenges in the field.

Author Information

Dustin Tran (Google Brain)

Dustin Tran is a research scientist at Google Brain. His research contributions examine the intersection of probability and deep learning, particularly in the areas of probabilistic programming, variational inference, giant models, and Bayesian neural networks. He completed his Ph.D. at Columbia under David Blei. He’s received awards such as the John M. Chambers Statistical Software award and the Google Ph.D. Fellowship in Machine Learning. He served as Area Chair at NeurIPS, ICML, ICLR, IJCAI, and AISTATS and organized "Approximate Inference" and "Uncertainty & Robustness" workshops at NeurIPS and UAI.

Balaji Lakshminarayanan (Google Deepmind)

Balaji Lakshminarayanan is a research scientist at Google Brain. Prior to that, he was a research scientist at DeepMind. He received his PhD from the Gatsby Unit, University College London where he worked with Yee Whye Teh. His recent research has focused on probabilistic deep learning, specifically, uncertainty estimation, out-of-distribution robustness and deep generative models. Notable contributions relevant to the tutorial include developing state-of-the-art methods for calibration under dataset shift (such as deep ensembles and AugMix) and showing that deep generative models do not always know what they don't know. He has co-organized several workshops on "Uncertainty and Robustness in deep learning" and served as Area Chair for NeurIPS, ICML, ICLR and AISTATS.

Jasper Snoek (Google Research, Brain team)

Jasper Snoek is a research scientist at Google Brain. His research has touched a variety of topics at the intersection of Bayesian methods and deep learning. He completed his PhD in machine learning at the University of Toronto. He subsequently held postdoctoral fellowships at the University of Toronto, under Geoffrey Hinton and Ruslan Salakhutdinov, and at the Harvard Center for Research on Computation and Society, under Ryan Adams. Jasper co-founded a Bayesian optimization focused startup, Whetlab, which was acquired by Twitter. He has served as an Area Chair for NeurIPS, ICML, AISTATS and ICLR, and organized a variety of workshops at ICML and NeurIPS.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors