Timezone: »

 
Reliability benchmarks for image segmentation
Estefany Kelly Buchanan · Michael Dusenberry · Jie Ren · Kevin Murphy · Balaji Lakshminarayanan · Dustin Tran

@
in Workshop on Distribution Shifts: Connecting Methods and Applications »
Event URL: https://openreview.net/forum?id=T6QZmBPlfv6 »

Recent work has shown the importance of reliability, where model performance is assessed under stress conditions pervasive in real-world deployment. In this work, we examine reliability tasks in the setting of semantic segmentation, a dense output problem that has typically only been evaluated using in-distribution predictive performance---for example, the mean intersection over union score on the Cityscapes validation set. To reduce the gap toward reliable deployment in the real world, we compile a benchmark involving existing (and newly constructed) distribution shifts and metrics. We evaluate current models and several baselines to determine how well segmentation models make robust predictions across multiple types of distribution shift and flag when they don’t know.

Author Information

Estefany Kelly Buchanan (Columbia University)
Michael Dusenberry (Google)
Jie Ren (Google Brain)
Kevin Murphy (Google)
Balaji Lakshminarayanan (Google Brain)

Balaji Lakshminarayanan is a research scientist at Google Brain. Prior to that, he was a research scientist at DeepMind. He received his PhD from the Gatsby Unit, University College London where he worked with Yee Whye Teh. His recent research has focused on probabilistic deep learning, specifically, uncertainty estimation, out-of-distribution robustness and deep generative models. Notable contributions relevant to the tutorial include developing state-of-the-art methods for calibration under dataset shift (such as deep ensembles and AugMix) and showing that deep generative models do not always know what they don't know. He has co-organized several workshops on "Uncertainty and Robustness in deep learning" and served as Area Chair for NeurIPS, ICML, ICLR and AISTATS.

Dustin Tran (Google Brain)

Dustin Tran is a research scientist at Google Brain. His research contributions examine the intersection of probability and deep learning, particularly in the areas of probabilistic programming, variational inference, giant models, and Bayesian neural networks. He completed his Ph.D. at Columbia under David Blei. He’s received awards such as the John M. Chambers Statistical Software award and the Google Ph.D. Fellowship in Machine Learning. He served as Area Chair at NeurIPS, ICML, ICLR, IJCAI, and AISTATS and organized "Approximate Inference" and "Uncertainty & Robustness" workshops at NeurIPS and UAI.

More from the Same Authors