Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
Anchit Jain · Rozhin Nobahari · Aristide Baratin · Stefano Sarao Mannelli
Keywords:
fairness
learning dynamics
stochastic gradient descent
online learning
analytical model
spurious correlation
Abstract
Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations. This paper explores the evolution of bias in a teacher-student setup modeling different data sub-populations with a Gaussian-mixture model, by providing an analytical description of the stochastic gradient descent dynamics of a linear classifier in thissetting. Our analysis reveals how different properties of sub-populations influence bias at different timescales, showing a shifting preference of the classifier during training. We empirically validate our results in more complex scenarios by training deeper networks on real datasets including CIFAR10, MNIST, and CelebA.
Chat is not available.
Successful Page Load