Counterfactual Situation Testing Eval

counterfactual-situation-testing-eval

Counterfactual Situation Testing: Uncovering Discrimination under Fairness given the Difference — Alvarez et al. (2023) (arXiv:2302.11944, 2023)

What this evaluates

Evaluates a fairness auditing framework's ability to detect individual discrimination in decision-making systems by comparing factual outcomes against counterfactual or similar-group outcomes. It probes whether protected attributes causally influence decisions beyond legitimate factors.

Datasets

• Synthetic Loan Application — total 5000; splits: test (5000); repo https://github.com/cc-jalvarez/counterfactual-situation-testing
• Law School Admissions — total 21790; splits: test (21790); repo https://github.com/cc-jalvarez/counterfactual-situation-testing

Metrics

• individual discrimination cases (primary) — range: percent
  • Percentage of complainants where the difference in prediction probability between the complainant and the test group ($\Delta p$) exceeds a minimum deviation threshold ($\tau=0.0$) and the threshold falls outside the 95% confidence interval of $\Delta p$.

Input / output format

Input: Factual feature vector $(X, A)$ for each applicant, along with the Structural Causal Model (SCM) equations and decision function $b()$ used to generate predictions.

Output: Binary discrimination label per instance (discriminated/not), aggregated as a count and percentage of total applicants.

Scoring recipe

discrimination_count = 0
for c in dataset:
    p_c = predict(c)
    p_t = predict(test_group_neighbors(c))
    delta_p = p_c - p_t
    ci = compute_confidence_interval(delta_p, alpha=0.05)
    if delta_p > 0.0 and 0.0 not in ci:
        discrimination_count += 1
return discrimination_count / len(dataset) * 100

Common pitfalls

• Confusing CST with standard Counterfactual Fairness (CF), which only compares an individual's factual vs. counterfactual outcome without group similarity constraints.
• Ignoring the confidence interval filter: cases with $\Delta p > 0$ but where the 95% CI includes 0 are statistically insignificant and should be excluded.
• Treating the synthetic and law school datasets as real-world benchmarks; they are procedurally generated using specific structural equations for illustration.

Evidence (verbatim from paper)

We define individual discrimination as $\Delta p>\tau$ (Def.[3.4]) for a single protected attribute. We still, though, demonstrate the use of confidence intervals (Def.[3.5]) and how it would affect the final results. Under $\alpha=5%$, we would reject these cases as individual discrimination claims with confidence level of 95% since the minimum deviation is covered by the CIs. Table 1 reports the Number (and %) of detected individual discrimination cases for the illustrative example based on gender.

Citation

@misc{alvarez2023counterfactual,
  title={Counterfactual Situation Testing: Uncovering Discrimination under Fairness given the Difference},
  author={Alvarez et al. (2023)},
  year={2023},
  note={arXiv:2302.11944}
}

• arXiv: 2302.11944