Timezone: »

 
Data Feedback Loops: Model-driven Amplification of Dataset Biases
Rohan Taori · Tatsunori Hashimoto

Sat Dec 03 01:20 PM -- 01:30 PM (PST) @
in Workshop on Distribution Shifts: Connecting Methods and Applications »
Event URL: https://openreview.net/forum?id=OJCNhcBFfp »

Datasets scraped from the internet have been critical to large-scale machine learning. Yet, its success puts the utility of future internet-derived datasets at potential risk, as model outputs begin to replace human annotations as a source of supervision. In this work, we formalize a system where interactions with one model are recorded as history and scraped as training data in the future. We then analyze its stability over time by tracking changes to a test-time bias statistic (e.g. gender bias of model predictions). We find that the degree of bias amplification is closely linked to whether the model’s outputs behave like samples from the training distribution, a behavior which we characterize and define as consistent calibration. Experiments in three conditional prediction scenarios – image classification, visual role-labeling, and language generation – demonstrate that models that exhibit a sampling-like behavior are more calibrated and thus more stable. Based on this insight, we propose an intervention to help calibrate and stabilize unstable feedback systems.

Author Information

Rohan Taori (Stanford University)
Rohan Taori

I like studying the foundations of machine learning in the context of real-world systems.

Tatsunori Hashimoto (Stanford)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors