Counterfactual Situation Testing: Fairness given the Difference

We present counterfactual situation testing (cfST), a new tool for detecting discrimination in datasets that operationalizes the Kohler-Hausmann Critique (KHC) of "fairness given the difference''. In situation testing (ST), like other discrimination analysis tools, the discrimination claim is recreated and thus tested by finding similar individuals to the one making the claim, the complainant $c$, and constructing a control group (what is) and a test group (what would have been if) of protected and non-protected individuals, respectively. ST builds both groups around $c$, which is wrong based on the KHC. Under cfST, we extend ST by constructing the control group around the complainant, which is the factual, and the test group around its counterfactual using the abduction, action, and prediction steps. We thus end up comparing control and test groups of not so similar individuals: one based on what we observe about $c$ versus one based on a hypothetical representation of $c$. By comparing these two different groups, we address the KHC and test for discrimination using a more meaningful causal interpretation of the protected attribute and its effects on all other attributes. We compare cfST to existing ST methods using synthetic data for a loan application process. The results show that cfST detects a higher number of discrimination cases than ST.