Fairprep Eval

fairprep-eval

Revisiting Pre-processing Group Fairness: A Modular Benchmarking Framework — Oldfield et al. (2025) (arXiv:2508.15193, 2025)

What this evaluates

This evaluation benchmarks pre-processing group fairness techniques on tabular datasets by measuring their impact on fairness metrics and downstream model performance. It probes whether bias mitigation methods improve group-level parity without significantly degrading predictive utility across varying decision thresholds.

Datasets

• Adult — total ?; splits: train (-1), test (-1)
• COMPAS — total ?; splits: train (-1), test (-1)
• German — total ?; splits: train (-1), test (-1)

Metrics

• disparate impact (primary) — range: [0, 1]
  • Ratio of the probability of a favorable outcome for the protected group to that of the unprotected group. Values closer to 1.0 indicate higher fairness.
• statistical parity difference — range: [-1, 1]
  • Difference in the probability of a favorable outcome between protected and unprotected groups. Values closer to 0.0 indicate higher fairness.
• balanced accuracy — range: [0, 1]
  • Average of recall obtained on each class, computed as (TPR + TNR) / 2.

Input / output format

Input: Tabular dataset containing feature columns, a target label column, and a protected attribute column. Data is optionally transformed by pre-processing techniques before being fed to a classifier.

Output: Predicted labels or probabilities from the trained model, along with computed fairness metrics (e.g., disparate impact, statistical parity difference) and performance metrics (e.g., balanced accuracy) evaluated across a sweep of classification thresholds.

Scoring recipe

def compute_metrics(y_true, y_pred, protected_attr, threshold=0.5):
    y_pred_binary = (y_pred >= threshold).astype(int)
    p_pos_protected = y_pred_binary[protected_attr == 1].mean()
    p_pos_unprotected = y_pred_binary[protected_attr == 0].mean()
    disparate_impact = p_pos_protected / p_pos_unprotected if p_pos_unprotected > 0 else 0.0
    spd = p_pos_protected - p_pos_unprotected
    tpr = (y_pred_binary[protected_attr == 1] == y_true[protected_attr == 1]).mean()
    tnr = (y_pred_binary[protected_attr == 0] == y_true[protected_attr == 0]).mean()
    bal_acc = (tpr + tnr) / 2.0
    return {'disparate_impact': disparate_impact, 'spd': spd, 'balanced_accuracy': bal_acc}

Common pitfalls

• Evaluating fairness at a single fixed threshold can be misleading, as metrics like equal opportunity and average odds difference fluctuate considerably across mid-range thresholds.
• Aggressively optimizing fairness metrics (e.g., via LFR) may destroy data fidelity or eliminate positive labels entirely, rendering the transformed dataset unrealistic for downstream tasks.

Evidence (verbatim from paper)

In the pre-processing stage, the framework computes pre-processing metrics on both the original and transformed datasets to assess the effect of bias mitigation before model training. In the benchmarking stage, each model is trained on the original and pre-processed datasets using a holdout validation scheme, and evaluated on both fairness and performance metrics. The resulting metrics are used to generate threshold-sensitive trade-off plots... Table[1] summarises fairness metrics computed on the original and processed datasets across three benchmark datasets and four pre-processing methods. Several consistent trends emerge. (1) RW improves group fairness across all datasets, achieving perfect disparate impact (1.0) and eliminating statistical parity difference (0.0)...

Citation

@misc{oldfield2025fairprep,
  title={Revisiting Pre-processing Group Fairness: A Modular Benchmarking Framework},
  author={Oldfield et al. (2025)},
  year={2025},
  note={arXiv:2508.15193}
}

• arXiv: 2508.15193