This workshop brings together domain experts and ML researchers working on mitigating distribution shifts in real-world applications.
Distribution shifts—where a model is deployed on a data distribution different from what it was trained on—pose significant robustness challenges in real-world ML applications. Such shifts are often unavoidable in the wild and have been shown to substantially degrade model performance in applications such as biomedicine, wildlife conservation, sustainable development, robotics, education, and criminal justice. For example, models can systematically fail when tested on patients from different hospitals or people from different demographics.
This workshop aims to convene a diverse set of domain experts and methods-oriented researchers working on distribution shifts. We are broadly interested in methods, evaluations and benchmarks, and theory for distribution shifts, and we are especially interested in work on distribution shifts that arise naturally in real-world application contexts. Examples of relevant topics include, but are not limited to:
- Examples of real-world distribution shifts in various application areas. We especially welcome applications that are not widely discussed in the ML research community, e.g., education, sustainable development, and conservation. We encourage submissions that characterize distribution shifts and their effects in real-world applications; it is not at all necessary to propose a solution that is algorithmically novel.
- Methods for improving robustness to distribution shifts. Relevant settings include domain generalization, domain adaptation, and subpopulation shifts, and we are interested in a wide range of approaches, from uncertainty estimation to causal inference to active data collection. We welcome methods that can work across a variety of shifts, as well as more domain-specific methods that incorporate prior knowledge on the types of shifts we wish to be robust on. We encourage evaluating these methods on real-world distribution shifts.
- Empirical and theoretical characterization of distribution shifts. Distribution shifts can vary widely in the way in which the data distribution changes, as well as the empirical trends they exhibit. What empirical trends do we observe? What empirical or theoretical frameworks can we use to characterize these different types of shifts and their effects? What kinds of theoretical settings capture useful components of real-world distribution shifts?
- Benchmarks and evaluations. We especially welcome contributions for subpopulation shifts, as they are underrepresented in current ML benchmarks. We are also interested in evaluation protocols that move beyond the standard assumption of fixed training and test splits -- for which applications would we need to consider other forms of shifts, such as streams of continually-changing data or feedback loops between models and data?
| Opening Remarks (Opening remarks for DistShift 2022) | |
| Domain Adaptation: Theory, Algorithms, and Open Library (Invited Talk) | |
| Machine-learning, distribution shifts and extrapolation in the Earth System (Invited Talk) | |
| Coffee Break | |
| The promises and pitfalls of CVAR (Invited Talk) | |
| Panel Discussion (In-person Panel Discussion) | |
| Lunch Break | |
| Poster Session | |
| First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains (Spotlight) | |
| Learning Invariant Representations under General Interventions on the Response (Spotlight) | |
| CAREER: Economic Prediction of Labor Sequence Data Under Distribution Shift (Spotlight) | |
| Tackling Distribution Shifts in Federated Learning with Superquantile Aggregation (Spotlight) | |
| Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization (Spotlight) | |
| Data Feedback Loops: Model-driven Amplification of Dataset Biases (Spotlight) | |
| Coffee Break | |
| External Validity: Framework, Design, and Analysis (Invited Talk) | |
| Bringing real-world data to bear in addressing distribution shifts: a sociolinguistically-informed analysis of ASR errors (Invited Talk) | |
| Geospatial Distribution Shifts in Ecology: Mapping the Urban Forest (Invited Talk) | |
| Closing Remarks (Closing remarks for DistShift 2022) | |
| A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods (Poster) | |
| A new benchmark for group distribution shifts in hand grasp regression for object manipulation. Can meta-learning raise the bar? (Poster) | |
| Instance norm improves meta-learning in class-imbalanced land cover classification (Poster) | |
| CUDA: Curriculum of Data Augmentation for Long-tailed Recognition (Poster) | |
| Sorted eigenvalue comparison $d_{\mathsf{Eig}}$: A simple alternative to $d_{\mathsf{FID}}$ (Poster) | |
| Undersampling is a Minimax Optimal Robustness Intervention in Nonparametric Classification (Poster) | |
| An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation (Poster) | |
| Characterizing Anomalies with Explainable Classifiers (Poster) | |
| Momentum-based Weight Interpolation of Strong Zero-Shot Models for Continual Learning (Poster) | |
| Performative Prediction with Neural Networks (Poster) | |
| Deconstructing Distributions: A Pointwise Framework of Learning (Poster) | |
| Train Offline, Test Online: A Real Robot Learning Benchmark (Poster) | |
| Using Interventions to Improve Out-of-Distribution Generalization of Text-Matching Systems (Poster) | |
| Domain Generalization for Robust Model-Based Offline Reinforcement Learning (Poster) | |
| Graph-Relational Distributionally Robust Optimization (Poster) | |
| Invariant Feature Subspace Recovery for Multi-Class Classification (Poster) | |
| Pushing the Accuracy-Fairness Tradeoff Frontier with Introspective Self-play (Poster) | |
| Class-wise Domain Generalization: A Novel Framework for Evaluating Distributional Shift (Poster) | |
| Diversity Boosted Learning for Domain Generalization with A Large Number of Domains (Poster) | |
| Federated Learning under Distributed Concept Drift (Poster) | |
| Sparse Mixture-of-Experts are Domain Generalizable Learners (Poster) | |
| Test-time adaptation with slot-centric models (Poster) | |
| RLSBench: A Large-Scale Empirical Study of Domain Adaptation Under Relaxed Label Shift (Poster) | |
| Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shifts (Poster) | |
| A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning (Poster) | |
| The Value of Out-of-distribution Data (Poster) | |
| Reliability benchmarks for image segmentation (Poster) | |
| Adaptive Pre-training of Language Models for Better Logical Reasoning (Poster) | |
| Dropout Disagreement: A Recipe for Group Robustness with Fewer Annotations (Poster) | |
| Multi-Domain Long-Tailed Learning by Augmenting Disentangled Representations (Poster) | |
| Meta-Adaptive Stock Movement Prediction with Two-Stage Representation Learning (Poster) | |
| Scale-conditioned Adaptation for Large Scale Combinatorial Optimization (Poster) | |
| Malign Overfitting: Interpolation and Invariance are Fundamentally at Odds (Poster) | |
| On the Abilities of Mathematical Extrapolation with Implicit Models (Poster) | |
| A Synthetic Limit Order Book Dataset for Benchmarking Forecasting Algorithms under Distributional Shift (Poster) | |
| A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias (Poster) | |
| Task Modeling: Approximating Multitask Predictions for Cross-Task Transfer (Poster) | |
| Generative Posterior Networks for Approximately Bayesian Epistemic Uncertainty Estimation (Poster) | |
| A Unified Framework for Comparing Learning Algorithms (Poster) | |
| Domain Generalization with Nuclear Norm Regularization (Poster) | |
| Memory bounds for continual learning (Poster) | |
| Out-of-Distribution Robustness via Targeted Augmentations (Poster) | |
| Surgical Fine-Tuning Improves Adaptation to Distribution Shifts (Poster) | |
| Characterising the Robustness of Reinforcement Learning for Continuous Control using Disturbance Injection (Poster) | |
| Tailored Overlap for Learning Under Distribution Shift (Poster) | |
| Mechanistic Lens on Mode Connectivity (Poster) | |
| Is Unsupervised Performance Estimation Impossible When Both Covariates and Labels shift? (Poster) | |
| First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains (Poster) | |
| DrML: Diagnosing and Rectifying Vision Models using Language (Poster) | |
| Empirical Study on Optimizer Selection for Out-of-Distribution Generalization (Poster) | |
| Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance (Poster) | |
| Robustmix: Improving Robustness by Regularizing the Frequency Bias of Deep Nets (Poster) | |
| Visual response inhibition for increased robustness of convolutional networks to distribution shifts (Poster) | |
| Learning with noisy labels using low-dimensional model trajectory (Poster) | |
| Evaluating the Impact of Geometric and Statistical Skews on Out-Of-Distribution Generalization Performance (Poster) | |
| An Invariant Learning Characterization of Controlled Text Generation (Poster) | |
| Relational Out-of-Distribution Generalization (Poster) | |
| Few Shot Generative Domain Adaptation Via Inference-Stage Latent Learning in GANs (Poster) | |
| Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization (Poster) | |
| Adaptive Sampling for Probabilistic Forecasting under Distribution Shift (Poster) | |
| A Learning Based Hypothesis Test for Harmful Covariate Shift (Poster) | |
| "Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts (Poster) | |
| Deep Class-Conditional Gaussians for Continual Learning (Poster) | |
| Benchmarking Robustness under Distribution Shift of Multimodal Image-Text Models (Poster) | |
| HICO-DET-SG and V-COCO-SG: New Data Splits to Evaluate Systematic Generalization in Human-Object Interaction Detection (Poster) | |
| Cross-Dataset Propensity Estimation for Debiasing Recommender Systems (Poster) | |
| Augmentation Consistency-guided Self-training for Source-free Domain Adaptive Semantic Segmentation (Poster) | |
| CAREER: Economic Prediction of Labor Sequence Data Under Distribution Shift (Poster) | |
| Impact of realistic properties of the point spread function on classification tasks to reveal a possible distribution shift (Poster) | |
| Mitigating Dataset Bias by Using Per-sample Gradient (Poster) | |
| Estimation of prediction error with known covariate shift (Poster) | |
| Reducing Forgetting in Federated Learning with Truncated Cross-Entropy (Poster) | |
| Learning to Extrapolate: A Transductive Approach (Poster) | |
| Strategy-Aware Contextual Bandits (Poster) | |
| Data Feedback Loops: Model-driven Amplification of Dataset Biases (Poster) | |
| Explanation Shift: Detecting distribution shifts on tabular data via the explanation space (Poster) | |
| Improving Domain Generalization with Interpolation Robustness (Poster) | |
| Enabling the Visualization of Distributional Shift using Shapley Values (Poster) | |
| Learning Invariant Representations under General Interventions on the Response (Poster) | |
| Theory and Algorithm for Batch Distribution Drift Problems (Poster) | |
| Frequency Shortcut Learning in Neural Networks (Poster) | |
| Preserving privacy with PATE for heterogeneous data (Poster) | |
| AdaME: Adaptive learning of multisource adaptationensembles (Poster) | |
| Transferability Between Regression Tasks (Poster) | |
| Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks (Poster) | |
| Tackling Distribution Shifts in Federated Learning with Superquantile Aggregation (Poster) | |
| Diversity through Disagreement for Better Transferability (Poster) | |
| Env-Aware Anomaly Detection: Ignore Style Changes, Stay True to Content! (Poster) | |
| Active Learning Over Multiple Domains in Natural Language Tasks (Poster) | |
| Few-Shot Learnable Augmentation for Financial Time Series Prediction under Distribution Shifts (Poster) | |
| Useful Confidence Measures: Beyond the Max Score (Poster) | |
| Multiple Modes for Continual Learning (Poster) | |
| A Closer Look at Novel Class Discovery from the Labeled Set (Poster) | |
| Out-of-Distribution Generalization Challenge in Dialog State Tracking (Poster) | |
| Toward domain generalized pruning by scoring out-of-distribution importance (Poster) | |
| Engineering Uncertainty Representations to Monitor Distribution Shifts (Poster) | |