Timezone: »

Are My Deep Learning Systems Fair? An Empirical Study of Fixed-Seed Training
Shangshu Qian · Viet Hung Pham · Thibaud Lutellier · Zeou Hu · Jungwon Kim · Lin Tan · Yaoliang Yu · Jiahao Chen · Sameena Shah

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @

Deep learning (DL) systems have been gaining popularity in critical tasks such as credit evaluation and crime prediction. Such systems demand fairness. Recent work shows that DL software implementations introduce variance: identical DL training runs (i.e., identical network, data, configuration, software, and hardware) with a fixed seed produce different models. Such variance could make DL models and networks violate fairness compliance laws, resulting in negative social impact. In this paper, we conduct the first empirical study to quantify the impact of software implementation on the fairness and its variance of DL systems. Our study of 22 mitigation techniques and five baselines reveals up to 12.6% fairness variance across identical training runs with identical seeds. In addition, most debiasing algorithms have a negative impact on the model such as reducing model accuracy, increasing fairness variance, or increasing accuracy variance. Our literature survey shows that while fairness is gaining popularity in artificial intelligence (AI) related conferences, only 34.4% of the papers use multiple identical training runs to evaluate their approach, raising concerns about their results’ validity. We call for better fairness evaluation and testing protocols to improve fairness and fairness variance of DL systems as well as DL research validity and reproducibility at large.

Author Information

Shangshu Qian (Purdue University)
Viet Hung Pham (University of Waterloo)
Thibaud Lutellier (University of Waterloo)
Zeou Hu (University of Waterloo)
Jungwon Kim (Purdue University)
Lin Tan (Purdue University)
Yaoliang Yu (Carnegie Mellon University)
Jiahao Chen (JPMorgan AI Research)

Jiahao Chen is a data science manager working in Capital One New York specializing in emerging technologies and university partnerships. He is currently the lead for the Banking in Explainable Algorithms Research (BEAR) group, focusing on FATES-related machine learning topics and their relation with banking regulations surrounding fair lending and explainability of credit decisioning to customers and regulators. Prior to joining Capital One in 2017, Jiahao was a Research Scientist at MIT CSAIL leading the Julia Lab, focusing on applications of the Julia programming language to various scientific data science problems and challenges in parallel computing and scientific computing.

Sameena Shah (J.P. Morgan Chase)

More from the Same Authors