Path Specific Fairness Eval

path-specific-fairness-eval

Path-Specific Counterfactual Fairness — Chiappa et al. (2018) (arXiv:1802.08139, 2018)

What this evaluates

Evaluates a model's ability to make predictions while removing the influence of a sensitive attribute along specific causal pathways, balancing predictive accuracy with path-specific counterfactual fairness constraints.

Datasets

• Berkeley Admission Dataset — total 4526; splits: train (3500), test (1026)
• UCI Adult Dataset — total 48842; splits: train (32561), test (16281)
• UCI German Credit Dataset — total 1000; splits: train (700), test (300)

Metrics

• fair accuracy (primary) — range: percent
  • Accuracy on the test set after applying counterfactual correction to remove unfair path effects along specified causal pathways.
• unfair accuracy — range: percent
  • Accuracy on the test set using the original model without fairness constraints or counterfactual correction.
• MMD — range: other
  • Maximum Mean Discrepancy between latent representations conditioned on the sensitive attribute for different groups, used as a penalization term to enforce fairness.

Input / output format

Input: Categorical and continuous features including a sensitive attribute (e.g., sex), along with other demographic and contextual variables.

Output: Binary or categorical prediction (e.g., admission decision, income >$50k, credit risk).

Scoring recipe

def compute_metrics(predictions, labels, sensitive_attr, latent_reprs):
    unfair_acc = np.mean(predictions == labels)
    fair_acc = np.mean(corrected_predictions == labels)
    mmd = maximum_mean_discrepancy(latent_reprs[sensitive_attr==0], latent_reprs[sensitive_attr==1])
    return unfair_acc, fair_acc, mmd

Common pitfalls

• The MMD penalization factor beta is schedule-dependent (0 for initial steps, then 1000 or 100), which drastically changes the trade-off between accuracy and fairness.
• Counterfactual correction is sometimes applied to both genders and sometimes only to the disadvantaged group, leading to different accuracy outcomes.
• The Berkeley dataset is artificially modified to include a direct bias path, which is not a standard benchmark split and requires careful reconstruction.

Evidence (verbatim from paper)

In Table 1, we show the unfair and fair accuracy on the test set at different stages of the training, together with the corresponding MMD values for H_m, H_l and H_r (× 10,000) for the UCI Adult dataset. Rows represent values after 5,000, 8,000, 15,000, and 20,000 training steps.

Citation

@misc{chiappa2018pathspecific,
  title={Path-Specific Counterfactual Fairness},
  author={Chiappa et al. (2018)},
  year={2018},
  note={arXiv:1802.08139}
}

• arXiv: 1802.08139