Fairpfneval

fairpfneval

FairPFN: Transformers Can do Counterfactual Fairness — Robertson et al. (2024) (arXiv:2407.05732, 2024)

What this evaluates

Evaluates a model's ability to mitigate the causal and counterfactual effects of protected attributes on predictions while maintaining predictive accuracy, without requiring explicit causal graph knowledge.

Datasets

• Synthetic Causal Case Studies — total ?; splits: test (-1)
• Law School Admissions — total ?; splits: test (-1)
• Adult Census Income — total ?; splits: test (-1)

Metrics

• Total Causal Effect (TCE/TeE) (primary) — range: other
  • Measures the total causal effect of the protected attribute on the prediction, computed using fitted structural causal models.
• 1-AUC — range: [0, 1]
  • Predictive error calculated as one minus the Area Under the ROC Curve.
• MAE — range: other
  • Mean Absolute Error between predictive distributions on real data ($\hat{Y}{real}$) and counterfactual data ($\hat{Y}{a\rightarrow a'}$).

Input / output format

Input: Observational feature vectors including protected attributes, provided without explicit causal graph or structural equation knowledge.

Output: Predicted values $\hat{Y}$ for real instances and counterfactual predictions $\hat{Y}_{a\rightarrow a'}$ for counterfactual instances.

Scoring recipe

# Compute predictive error
auc = compute_auc(predictions, ground_truth)
error_1_auc = 1 - auc

# Compute causal effect (TCE/TeE)
scm = fit_scm(data)
tce = compute_total_causal_effect(scm, protected_attribute, predictions)

# Compute counterfactual MAE
mae = mean_absolute_error(predictions_real, predictions_counterfactual)

Common pitfalls

• Assuming the model requires explicit causal graph knowledge during inference (the paper explicitly states it receives no such information).
• Expecting the model to output a random classifier when data is heavily biased; FairPFN retains non-causal noise terms to preserve predictive signal.
• Confusing Total Causal Effect (TCE) with direct or indirect effects; the evaluation focuses on the total effect removal.

Evidence (verbatim from paper)

In this section, we evaluate the performance of FairPFN on our benchmark of synthetic and real-world scenarios, with the key message that FairPFN removes the causal and counterfactual effect of protected attributes without any knowledge of the causal model. ... We also evaluate FairPFN on the Law School Admissions and Adult Census Income datasets, using causal models fit to the structures posed in Figure 6 to measure the TeE and MAE. We note again that in evaluation FairPFN receives no information about the causal graphs or models. In Figure 7, we also measure the MAE between the predictive distributions on the real and counterfactual datasets, $\hat{Y}{real}$ and $\hat{Y}{a\rightarrow a'}$.

Citation

@misc{robertson2024fairpf,
  title={FairPFN: Transformers Can do Counterfactual Fairness},
  author={Robertson et al. (2024)},
  year={2024},
  note={arXiv:2407.05732}
}

• arXiv: 2407.05732