Hmda Fairness Eval

hmda-fairness-eval

Overcoming Fairness Trade-offs via Pre-processing: A Causal Perspective — Leininger et al. (2025) (arXiv:2501.14710, 2025)

What this evaluates

Evaluates the trade-off between predictive performance and group fairness when applying causal pre-processing to approximate an unbiased data distribution. It probes whether debiasing techniques can simultaneously satisfy multiple fairness constraints without degrading model accuracy.

Datasets

• HMDA (Wisconsin, 2022) — total 83808; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the model's ability to rank positive (loan approved) instances higher than negative ones across all classification thresholds.
• Fairness metrics (Demographic Parity) — range: percent
  • Difference in approval rates between the protected group (Black) and unprotected group (White). Evaluated as a percentage constraint, with the paper targeting 100% parity (0% difference).

Input / output format

Input: Features: loan amount, loan purpose, debt ratio, age, gender. Protected attribute: race (Black vs. non-Hispanic White).

Output: Binary loan approval decision (1 = granted, 0 = not granted) or predicted probability.

Scoring recipe

def score(predictions, gold, protected_mask):
    auc = roc_auc_score(gold, predictions)
    approval_rate_protected = mean(predictions[protected_mask])
    approval_rate_unprotected = mean(predictions[~protected_mask])
    fairness_diff = abs(approval_rate_protected - approval_rate_unprotected)
    return {'AUC': auc, 'Fairness_Diff': fairness_diff}

Common pitfalls

• Evaluating performance on real-world data instead of pre-processed data, which masks the true impact of bias mitigation and overestimates accuracy.
• Ignoring the severe class imbalance (protected group is only ~5.2% of the dataset) when interpreting fairness-performance trade-offs.
• Assuming higher AUC on real data indicates better model quality, whereas the paper argues it merely reflects entrenched historical bias.

Evidence (verbatim from paper)

For the real world HMDA data in Figure 4a, we again observe the common trade-off between fairness and performance, where an increase in fairness from 80% (fairness of the unconstraint model) to 100% is accompanied by a decrease in performance. Regarding performance, we observe that models trained and evaluated on real, adapted, and warped data all achieve similarly high AUCs.

Citation

@misc{leininger2025overcoming,
  title={Overcoming Fairness Trade-offs via Pre-processing: A Causal Perspective},
  author={Leininger et al. (2025)},
  year={2025},
  note={arXiv:2501.14710}
}

• arXiv: 2501.14710