Fairpfn Causal Fairness Eval

fairpfn-causal-fairness-eval

FairPFN: A Tabular Foundation Model for Causal Fairness — Robertson et al. (2025) (arXiv:2506.07049, 2025)

What this evaluates

Evaluates a model's ability to remove the causal influence of protected attributes from predictions while maintaining predictive accuracy. It probes counterfactual fairness and causal effect removal on both synthetically generated causal graphs and real-world tabular datasets.

Datasets

• Synthetic Causal Case Studies — total 600; splits: test (600)
• Law School Admissions — total ?; splits: test (-1)
• Adult Census Income — total ?; splits: test (-1)

Metrics

• ATE (primary) — range: [-1, 1]
  • Average Treatment Effect: the mean difference in predicted outcomes between factual and counterfactual protected attribute values, measuring causal effect removal.
• 1-AUC — range: [0, 1]
  • Predictive error calculated as 1 minus the Area Under the Receiver Operating Characteristic Curve.
• AE — range: [0, 1]
  • Absolute Error: the mean absolute difference between predictions on observational and counterfactual inputs, evaluated at the individual sample level.

Input / output format

Input: Tabular feature matrix X, protected attribute A, and target Y for training. For evaluation, observational inputs (X, A) and counterfactual inputs (X, A→a′) where the protected attribute is flipped.

Output: Predicted outcome probabilities or class labels Ŷ.

Scoring recipe

def score(predictions, gold, protected_attr, cf_predictions):
    # ATE: causal effect removal
    ate = abs(mean(predictions[protected_attr == 0]) - mean(predictions[protected_attr == 1]))
    # 1-AUC: predictive error
    auc = roc_auc_score(gold, predictions)
    error = 1 - auc
    # AE: counterfactual fairness
    ae = mean(abs(predictions - cf_predictions))
    return {"ATE": ate, "1-AUC": error, "AE": ae}

Common pitfalls

• ATE measures causal effect removal rather than statistical parity; lower values indicate better fairness.
• Synthetic benchmarks use known ground-truth causal weights, making baselines with causal access artificially strong compared to real-world settings where causal graphs must be inferred.
• Counterfactual fairness is evaluated at the individual sample level using AE, not just at the group level.

Evidence (verbatim from paper)

We evaluate FairPFN’s efficacy in causal effect removal by analyzing box plots depicting the median, interquartile range (IQR), and average treatment effect (ATE) of predictions, compared to baseline predictive models that also do not access causal information (Figure [4]). ... Next, we evaluate the counterfactual fairness of FairPFN on real-world datasets as introduced in Section [3], noting that the following analysis is conducted at the individual sample level, rather than at the dataset level. Figure [7] illustrates the distribution of Absolute Error (AE) achieved by FairPFN and baselines that do not have access to causal information.

Citation

@misc{robertson2025fairpfn,
  title={FairPFN: A Tabular Foundation Model for Causal Fairness},
  author={Robertson et al. (2025)},
  year={2025},
  note={arXiv:2506.07049}
}

• arXiv: 2506.07049