Timezone: »
Accurately estimating and explaining an ML model’s performance on new datasets is increasingly critical in reliable ML model deployment. With no labels on the new datasets, performance estimation paradigms often assume either covariate shift or label shift, and thus lead to poor estimation accuracy when the assumptions are broken. Is unsupervised performance monitoring really impossible when both covariates and labels shift? In this paper, we give a negative answer. To do so, we introduce Sparse Joint Shift (SJS), a new distribution shift model considering the shift of labels and a few features. We characterize the mathematical conditions under which SJS is identifiable. This shows that unsupervised performance monitoring is indeed feasible when a few features and labels shift. In addition, we propose SEES, an algorithmic framework for performance estimation under SJS. Preliminary experiments show the superior estimation performance of SEES over existing paradigms. This opens the door to tackling the joint shift of both covariates and labels without observing new datasets’ labels.
Author Information
Lingjiao Chen (Stanford University)
Matei Zaharia (Stanford University)
James Zou (Stanford)
More from the Same Authors
-
2022 : Predicting Immune Escape with Pretrained Protein Language Model Embeddings »
Kyle Swanson · Howard Chang · James Zou -
2022 : Data-driven subgroup identification for linear regression »
Zachary Izzo · Ruishan Liu · James Zou -
2022 : DrML: Diagnosing and Rectifying Vision Models using Language »
Yuhui Zhang · Jeff Z. HaoChen · Shih-Cheng Huang · Kuan-Chieh Wang · James Zou · Serena Yeung -
2022 : Provable Re-Identification Privacy »
Zachary Izzo · Jinsung Yoon · Sercan Arik · James Zou -
2022 : Recommendation for New Drugs with Limited Prescription Data »
Zhenbang Wu · Huaxiu Yao · Zhe Su · David Liebovitz · Lucas Glass · James Zou · Chelsea Finn · Jimeng Sun -
2023 : Generative AI for designing and validating easily synthesizable and structurally novel antibiotics »
Kyle Swanson · Gary Liu · Denise Catacutan · James Zou · Jonathan Stokes -
2023 : Analyzing ChatGPT’s Behavior Shifts Over Time »
Lingjiao Chen · Matei A Zaharia · James Zou -
2023 : Navigating Dataset Documentation in ML: A Large-Scale Analysis of Dataset Cards on Hugging Face »
Xinyu Yang · Weixin Liang · James Zou -
2023 : A Theoretical Study of Dataset Distillation »
Zachary Izzo · James Zou -
2023 Poster: TWIGMA: A dataset of AI-Generated Images with Metadata From Twitter »
Yiqun Chen · James Zou -
2023 Poster: Factorized Contrastive Learning: Going Beyond Multi-view Redundancy »
Paul Pu Liang · Zihao Deng · Martin Q. Ma · James Zou · Louis-Philippe Morency · Ruslan Salakhutdinov -
2023 Poster: Beyond Confidence: Reliable Models Should Also Consider Atypicality »
Mert Yuksekgonul · Linjun Zhang · James Zou · Carlos Guestrin -
2023 Poster: OpenDataVal: a Unified Benchmark for Data Valuation »
Kevin Jiang · Weixin Liang · James Zou · Yongchan Kwon -
2023 Poster: DataPerf: Benchmarks for Data-Centric AI Development »
Mark Mazumder · Colby Banbury · Xiaozhe Yao · Bojan Karlaš · William Gaviria Rojas · Sudnya Diamos · Greg Diamos · Lynn He · Alicia Parrish · Hannah Rose Kirk · Jessica Quaye · Charvi Rastogi · Douwe Kiela · David Jurado · David Kanter · Rafael Mosquera · Will Cukierski · Juan Ciro · Lora Aroyo · Bilge Acun · Lingjiao Chen · Mehul Raje · Max Bartolo · Evan Sabri Eyuboglu · Amirata Ghorbani · Emmett Goodman · Addison Howard · Oana Inel · Tariq Kane · Christine R. Kirkpatrick · D. Sculley · Tzu-Sheng Kuo · Jonas Mueller · Tristan Thrush · Joaquin Vanschoren · Margaret Warren · Adina Williams · Serena Yeung · Newsha Ardalani · Praveen Paritosh · Ce Zhang · James Zou · Carole-Jean Wu · Cody Coleman · Andrew Ng · Peter Mattson · Vijay Janapa Reddi -
2022 : An Electrocardiogram-Based Risk Score for Cardiovascular Mortality »
John Hughes · David Ouyang · Pierre Elias · James Zou · Euan Ashley · Marco Perez -
2022 : An Electrocardiogram-Based Risk Score for Cardiovascular Mortality »
John Hughes · David Ouyang · Pierre Elias · James Zou · Euan Ashley · Marco Perez -
2022 Poster: Estimating and Explaining Model Performance When Both Covariates and Labels Shift »
Lingjiao Chen · Matei Zaharia · James Zou -
2022 Poster: SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis »
Roxana Daneshjou · Mert Yuksekgonul · Zhuo Ran Cai · Roberto Novoa · James Zou -
2022 Poster: HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions »
Lingjiao Chen · Zhihua Jin · Evan Sabri Eyuboglu · Christopher Ré · Matei Zaharia · James Zou -
2022 Poster: Uncalibrated Models Can Improve Human-AI Collaboration »
Kailas Vodrahalli · Tobias Gerstenberg · James Zou -
2022 Poster: C-Mixup: Improving Generalization in Regression »
Huaxiu Yao · Yiping Wang · Linjun Zhang · James Zou · Chelsea Finn -
2022 Poster: Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning »
Weixin Liang · Yuhui Zhang · Yongchan Kwon · Serena Yeung · James Zou -
2022 Poster: WeightedSHAP: analyzing and improving Shapley based feature attributions »
Yongchan Kwon · James Zou -
2021 Poster: Adversarial Training Helps Transfer Learning via Better Representations »
Zhun Deng · Linjun Zhang · Kailas Vodrahalli · Kenji Kawaguchi · James Zou -
2020 Session: Orals & Spotlights Track 02: COVID/Health/Bio Applications »
Tristan Naumann · James Zou -
2019 Poster: Making AI Forget You: Data Deletion in Machine Learning »
Antonio Ginart · Melody Guan · Gregory Valiant · James Zou -
2019 Spotlight: Making AI Forget You: Data Deletion in Machine Learning »
Antonio Ginart · Melody Guan · Gregory Valiant · James Zou -
2017 Workshop: Machine Learning in Computational Biology »
James Zou · Anshul Kundaje · Gerald Quon · Nicolo Fusi · Sara Mostafavi -
2017 Poster: NeuralFDR: Learning Discovery Thresholds from Hypothesis Features »
Fei Xia · Martin J Zhang · James Zou · David Tse