Timezone: »

 
Poster
JAWS: Auditing Predictive Uncertainty Under Covariate Shift
Drew Prinster · Anqi Liu · Suchi Saria

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #941
in Poster Session 2 »

We propose \textbf{JAWS}, a series of wrapper methods for distribution-free uncertainty quantification tasks under covariate shift, centered on the core method \textbf{JAW}, the \textbf{JA}ckknife+ \textbf{W}eighted with data-dependent likelihood-ratio weights. JAWS also includes computationally efficient \textbf{A}pproximations of JAW using higher-order influence functions: \textbf{JAWA}. Theoretically, we show that JAW relaxes the jackknife+'s assumption of data exchangeability to achieve the same finite-sample coverage guarantee even under covariate shift. JAWA further approaches the JAW guarantee in the limit of the sample size or the influence function order under common regularity assumptions. Moreover, we propose a general approach to repurposing predictive interval-generating methods and their guarantees to the reverse task: estimating the probability that a prediction is erroneous, based on user-specified error criteria such as a safe or acceptable tolerance threshold around the true label. We then propose \textbf{JAW-E} and \textbf{JAWA-E} as the repurposed proposed methods for this \textbf{E}rror assessment task. Practically, JAWS outperform state-of-the-art predictive inference baselines in a variety of biased real world data sets for interval-generation and error-assessment predictive uncertainty auditing tasks.

Keywords: uncertainty quantification · jackknife+ · covariate shift · conformal prediction · Influence Functions · auditing · error assessment
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Drew Prinster (Johns Hopkins University)
Anqi Liu (JHU)
Suchi Saria (Johns Hopkins University)

Suchi Saria is an assistant professor of computer science, health policy and statistics at Johns Hopkins University. Her research interests are in statistical machine learning and computational healthcare. Specifically, her focus is in designing novel data-driven computing tools for optimizing decision-making. Her work is being used to drive electronic surveillance for reducing adverse events in the inpatient setting and individualize disease management in chronic diseases. She received her PhD from Stanford University with Prof. Daphne Koller. Her work has received recognition in the form of two cover articles in Science Translational Medicine (2010, 2015), paper awards by the the Association for Uncertainty in Artificial Intelligence (2007) and the American Medical Informatics Association (2011), an Annual Scientific Award by the Society of Critical Care Medicine (2014), a Rambus Fellowship (2004-2010), an NSF Computing Innovation fellowship (2011), and competitive awards from the Gordon and Betty Moore Foundation (2013), and Google Research (2014). In 2015, she was selected by the IEEE Intelligent Systems to the ``AI's 10 to Watch'' list. In 2016, she was selected as a DARPA Young Faculty awardee and to Popular Science's ``Brilliant 10’’.

More from the Same Authors