# Cafp Fairness Eval

> Evaluates whether a post-processing framework can reduce group-level disparities in predictions while maintaining predictive accuracy. It probes a model's ability to balance fairness constraints (demographic parity and equalized odds) against standard classification performance across multiple benchmark datasets. Use when the user wants to benchmark on Adult Income (UCI), COMPAS Recidivism, German Credit, or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/cafp-fairness-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/cafp-fairness-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/cafp-fairness-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/cafp-fairness-eval

---


# cafp-fairness-eval

> CAFP: A Post-Processing Framework for Group Fairness via Counterfactual Model Averaging — Arévalo et al. (2026) (arXiv:2604.07009, 2026)

## What this evaluates

Evaluates whether a post-processing framework can reduce group-level disparities in predictions while maintaining predictive accuracy. It probes a model's ability to balance fairness constraints (demographic parity and equalized odds) against standard classification performance across multiple benchmark datasets.

## Datasets

- **Adult Income (UCI)** — total ?; splits: train (-1), test (-1)
- **COMPAS Recidivism** — total ?; splits: train (-1), test (-1)
- **German Credit** — total ?; splits: train (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances.
- `Average Odds Difference (AOD)` — range: [0, 1]
  - Average of the absolute differences in the true positive rate (TPR) and false positive rate (FPR) between groups: 0.5 * (|TPR_0 - TPR_1| + |FPR_0 - FPR_1|).
- `Demographic Parity Difference (DPD)` — range: [0, 1]
  - Absolute difference in positive prediction rates across groups: |P(Ŷ=1|A=0) - P(Ŷ=1|A=1)|.

## Input / output format

**Input**: Standardized feature vectors, binary classification labels, and binary-encoded protected attributes. For CAFP, the base model's raw predictions (or probabilities) are also required to generate counterfactual predictions.

**Output**: Adjusted binary predictions (or probabilities) after counterfactual averaging, along with computed fairness and accuracy metrics reported with 95% confidence intervals and standard deviations.

## Scoring recipe

```python
def compute_aod(y_true, y_pred, protected_attr):
    tpr_0 = (y_true[protected_attr==0] & y_pred[protected_attr==0]).sum() / (protected_attr==0).sum()
    fpr_0 = (~y_true[protected_attr==0] & y_pred[protected_attr==0]).sum() / (protected_attr==0).sum()
    tpr_1 = (y_true[protected_attr==1] & y_pred[protected_attr==1]).sum() / (protected_attr==1).sum()
    fpr_1 = (~y_true[protected_attr==1] & y_pred[protected_attr==1]).sum() / (protected_attr==1).sum()
    return 0.5 * (abs(tpr_0 - tpr_1) + abs(fpr_0 - fpr_1))

def compute_dpd(y_pred, protected_attr):
    p_0 = y_pred[protected_attr==0].mean()
    p_1 = y_pred[protected_attr==1].mean()
    return abs(p_0 - p_1)
```

## Common pitfalls

- Protected attributes are binarized (e.g., age split at 25), which may obscure nuanced demographic disparities.
- CAFP requires access to base model predictions on counterfactual instances, which can be computationally expensive or require model-specific assumptions.
- Fairness metrics are evaluated across repeated runs and reported with 95% confidence intervals and standard deviations, not as single-point estimates.

## Evidence (verbatim from paper)

> We report the following metrics to evaluate performance and fairness: Accuracy: Overall predictive performance. Average Odds Difference (AOD): Average of the absolute differences in the true positive rate (TPR) and false positive rate (FPR) between groups defined by a protected attribute. Demographic Parity Difference (DPD): Absolute difference in positive prediction rates across groups.

## Citation

```bibtex
@misc{arevalo2026cafp,
  title={CAFP: A Post-Processing Framework for Group Fairness via Counterfactual Model Averaging},
  author={Arévalo et al. (2026)},
  year={2026},
  note={arXiv:2604.07009}
}
```

- arXiv: 2604.07009