Fair Credit Scoring Eval

fair-credit-scoring-eval

Fairness in Credit Scoring: Assessment, Implementation and Profit Implications — Kozodoi et al. (2021) (arXiv:2103.01907, 2021)

What this evaluates

This benchmark evaluates the trade-off between algorithmic fairness and financial profitability in credit scoring models. It probes how well various fairness-aware preprocessing, in-processing, and post-processing techniques maintain predictive accuracy and demographic parity while minimizing economic loss for lenders.

Datasets

• german — total 1000; splits: train (600), test (400)
• bene — total 3123; splits: train (1873), test (1250)
• taiwan — total 23531; splits: train (14118), test (9413)
• uk — total 30000; splits: train (18000), test (12000)
• pakdd — total 50000; splits: train (30000), test (20000)
• gmsc — total 150000; splits: train (90000), test (60000)
• homecredit — total 307511; splits: train (184506), test (123005)

Metrics

• Profitability (Profit per EUR) (primary) — range: percent
  • Normalized expected profit per EUR issued, computed via the EMP criterion. It integrates over the cost of default (B) and opportunity cost/benefit (C=ROI=0.2664) using predicted cumulative density functions for good and bad risks at a cutoff τ. Base scenario is rejecting all applications.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the model's discriminatory ability across all classification thresholds.
• Fairness (Independence, Separation, Sufficiency) — range: [0, 1]
  • Independence measures demographic parity (prediction independent of sensitive attribute). Separation measures equalized odds (true/false positive rates equal across groups). Sufficiency measures calibration within groups (positive predictive value equal across groups).

Input / output format

Input: Tabular loan applicant features and loan characteristics. Target is binary (1=repaid, 0=default). Sensitive attribute is age group split at 25 years (<25 vs ≥25).

Output: Continuous risk score or binary approval decision (approve/reject) derived via a threshold τ.

Scoring recipe

def compute_metrics(y_true, y_scores, sensitive, priors, B_dist, C=0.2664):
    # Profit: Apply Eq 16 using predicted CDFs F0(tau), F1(tau)
    # Integrate over B distribution: [C*(pi1*(1-F1) - pi1*F1) - B*pi0*(1-F0)] * f(B) dB
    profit = compute_emp_profit(y_scores, y_true, priors, B_dist, C)
    # AUC: Standard ROC-AUC
    auc = roc_auc_score(y_true, y_scores)
    # Fairness: Compute Independence, Separation, Sufficiency across sensitive groups
    fairness = compute_fairness_metrics(y_true, y_scores, sensitive)
    return {'profit': profit, 'auc': auc, 'fairness': fairness}

Common pitfalls

• The profit baseline is normalized to 'rejecting all applications', not the standard 'accepting all' or random baseline, which drastically changes the magnitude and sign of profit differences.
• The sensitive attribute is age (<25 vs ≥25), not race or gender, and the threshold of 25 is empirically derived from disparate impact analysis, not a standard demographic split.
• Aggregation differs by processor type: pre/post-processors average over 140 runs (7 datasets × 5 folds × 4 classifiers), while in-processors average over only 35 runs (7 datasets × 5 folds).

Evidence (verbatim from paper)

Fairness processors and benchmarks are evaluated on the test set using multiple performance metrics. First, we measure the profitability of a scorecard by computing profit per EUR issued by a financial institution. ... Apart from estimating the profitability of each fairness processor, we also compute the area under the ROC curve (AUC), which is a widely used indicator of the discriminatory ability of a scoring model. In addition, we evaluate fairness by measuring independence, separation and sufficiency.

Citation

@misc{kozodoi2021fairness,
  title={Fairness in Credit Scoring: Assessment, Implementation and Profit Implications},
  author={Kozodoi et al. (2021)},
  year={2021},
  note={arXiv:2103.01907}
}

• arXiv: 2103.01907