Cafp Fairness Eval

cafp-fairness-eval

CAFP: A Post-Processing Framework for Group Fairness via Counterfactual Model Averaging — Arévalo et al. (2026) (arXiv:2604.07009, 2026)

What this evaluates

Evaluates whether a post-processing framework can reduce group-level disparities in predictions while maintaining predictive accuracy. It probes a model's ability to balance fairness constraints (demographic parity and equalized odds) against standard classification performance across multiple benchmark datasets.

Datasets

• Adult Income (UCI) — total ?; splits: train (-1), test (-1)
• COMPAS Recidivism — total ?; splits: train (-1), test (-1)
• German Credit — total ?; splits: train (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.
• Average Odds Difference (AOD) — range: [0, 1]
  • Average of the absolute differences in the true positive rate (TPR) and false positive rate (FPR) between groups: 0.5 * (|TPR_0 - TPR_1| + |FPR_0 - FPR_1|).
• Demographic Parity Difference (DPD) — range: [0, 1]
  • Absolute difference in positive prediction rates across groups: |P(Ŷ=1|A=0) - P(Ŷ=1|A=1)|.

Input / output format

Input: Standardized feature vectors, binary classification labels, and binary-encoded protected attributes. For CAFP, the base model's raw predictions (or probabilities) are also required to generate counterfactual predictions.

Output: Adjusted binary predictions (or probabilities) after counterfactual averaging, along with computed fairness and accuracy metrics reported with 95% confidence intervals and standard deviations.

Scoring recipe

def compute_aod(y_true, y_pred, protected_attr):
    tpr_0 = (y_true[protected_attr==0] & y_pred[protected_attr==0]).sum() / (protected_attr==0).sum()
    fpr_0 = (~y_true[protected_attr==0] & y_pred[protected_attr==0]).sum() / (protected_attr==0).sum()
    tpr_1 = (y_true[protected_attr==1] & y_pred[protected_attr==1]).sum() / (protected_attr==1).sum()
    fpr_1 = (~y_true[protected_attr==1] & y_pred[protected_attr==1]).sum() / (protected_attr==1).sum()
    return 0.5 * (abs(tpr_0 - tpr_1) + abs(fpr_0 - fpr_1))

def compute_dpd(y_pred, protected_attr):
    p_0 = y_pred[protected_attr==0].mean()
    p_1 = y_pred[protected_attr==1].mean()
    return abs(p_0 - p_1)

Common pitfalls

• Protected attributes are binarized (e.g., age split at 25), which may obscure nuanced demographic disparities.
• CAFP requires access to base model predictions on counterfactual instances, which can be computationally expensive or require model-specific assumptions.
• Fairness metrics are evaluated across repeated runs and reported with 95% confidence intervals and standard deviations, not as single-point estimates.

Evidence (verbatim from paper)

We report the following metrics to evaluate performance and fairness: Accuracy: Overall predictive performance. Average Odds Difference (AOD): Average of the absolute differences in the true positive rate (TPR) and false positive rate (FPR) between groups defined by a protected attribute. Demographic Parity Difference (DPD): Absolute difference in positive prediction rates across groups.

Citation

@misc{arevalo2026cafp,
  title={CAFP: A Post-Processing Framework for Group Fairness via Counterfactual Model Averaging},
  author={Arévalo et al. (2026)},
  year={2026},
  note={arXiv:2604.07009}
}

• arXiv: 2604.07009