`

Timezone: »

 
Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning
Zachary Nado · Neil Band · Mark Collier · Josip Djolonga · Mike Dusenberry · Sebastian Farquhar · Qixuan Feng · Angelos Filos · Marton Havasi · Rodolphe Jenatton · Ghassen Jerfel · Jeremiah Liu · Zelda Mariet · Jeremy Nixon · Shreyas Padhy · Jie Ren · Tim G. J. Rudner · Yeming Wen · Florian Wenzel · Kevin Murphy · D. Sculley · Balaji Lakshminarayanan · Jasper Snoek · Yarin Gal · Dustin Tran
Event URL: https://openreview.net/forum?id=ehUzL3tOKEP »

High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ability to compare techniques for improving these estimates is therefore very important for research and practice alike. Yet, competitive comparisons of methods are often lacking due to a range of reasons, including: compute availability for extensive tuning, incorporation of sufficiently many baselines, and concrete documentation for reproducibility. In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-ofthe-art deep learning methods on a variety of tasks. As of this writing, the collection spans 19 methods across 9 tasks, each with at least 5 metrics. Each baseline is a self-contained experiment pipeline with easily reusable and extendable components. Our goal is to provide immediate starting points for experimentation with new methods or applications. Additionally we provide model checkpoints, experiment outputs as Python notebooks, and leaderboards for comparing results. https://github.com/google/uncertainty-baselines

Author Information

Zachary Nado (Google Inc.)
Neil Band (University of Oxford)
Mark Collier (Google)
Josip Djolonga (Google Research, Brain Team)
Mike Dusenberry (Google Research, Brain team)
Sebastian Farquhar (University of Oxford)
Qixuan Feng (University of Oxford)
Angelos Filos (University of Oxford)
Marton Havasi (University of Cambridge)
Rodolphe Jenatton (Amazon)
Ghassen Jerfel (Duke University)
Jeremiah Liu (Google Research / Harvard)
Zelda Mariet (Google Brain)
Jeremy Nixon (Google Brain)
Shreyas Padhy (Google)
Jie Ren (Google Brain)
Tim G. J. Rudner (University of Oxford)
Yeming Wen (University of Texas, Austin)
Florian Wenzel (---)
Kevin Murphy (Google)
D. Sculley (Google Research)
Balaji Lakshminarayanan (Google Brain)
Jasper Snoek (University of Toronto)
Yarin Gal (University of Oxford)
Dustin Tran (Google Brain)

More from the Same Authors