Fairx Gcig Eval

fairx-gcig-eval

Procedural Fairness via Group Counterfactual Explanation — Popoola et al. (2026) (arXiv:2603.11140, 2026)

What this evaluates

Evaluates whether a model maintains predictive utility while achieving procedural fairness (explanation invariance across protected groups) and outcome fairness. It probes the alignment between equalized odds and group-level feature attribution consistency.

Datasets

• Adult — total 4522; splits: train (-1), val (-1), test (-1)
• German Credit — total 1000; splits: train (-1), val (-1), test (-1)
• COMPAS — total 6172; splits: train (-1), val (-1), test (-1)
• Bank Marketing — total 45211; splits: train (-1), val (-1), test (-1)

Metrics

• GCIG (primary) — range: other
  • Group Counterfactual Integrated Gradients. Computes Integrated Gradients for each instance relative to group-conditional baselines and measures the cross-group variation in feature attributions. Lower values indicate higher explanation invariance.
• EO Gap — range: other
  • Equalized Odds gap. Measures the absolute difference in True Positive Rates and False Positive Rates across protected groups. Lower values indicate better outcome fairness.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall. Higher values indicate better predictive utility.

Input / output format

Input: Tabular dataset with standardized continuous features, one-hot encoded categorical features, a binary target label, and a protected attribute column.

Output: Binary predictions for utility/fairness metrics, and per-instance feature attribution vectors (Integrated Gradients) for procedural fairness computation.

Scoring recipe

def compute_metrics(preds, gold, groups, model, inputs):
    tp = sum((p == 1) & (g == 1) for p, g in zip(preds, gold))
    fp = sum((p == 1) & (g == 0) for p, g in zip(preds, gold))
    fn = sum((p == 0) & (g == 1) for p, g in zip(preds, gold))
    f1 = 2 * tp / (2 * tp + fp + fn) if (2 * tp + fp + fn) > 0 else 0.0
    tpr_a = sum((p == 1) & (g == 1) & (gr == 'A') for p, g, gr in zip(preds, gold, groups)) / max(sum((g == 1) & (gr == 'A') for g, gr in zip(gold, groups)), 1)
    tpr_b = sum((p == 1) & (g == 1) & (gr == 'B') for p, g, gr in zip(preds, gold, groups)) / max(sum((g == 1) & (gr == 'B') for g, gr in zip(gold, groups)), 1)
    eo_gap = abs(tpr_a - tpr_b)
    return f1, eo_gap

Common pitfalls

• Assuming low EO Gap guarantees low GCIG; the paper shows only weak correlation (r=0.244) and >94% unexplained variance.
• Using a single global baseline for Integrated Gradients instead of group-conditional baselines, which GCIG explicitly requires.
• Failing to stratify data splits by both label and protected attribute, which can bias disparity estimates.

Evidence (verbatim from paper)

To address RQ1, Table 4.2 summarizes predictive performance (F1), outcome fairness (EO gap), and procedural fairness (GCIG) across four datasets. Results are reported as mean ± standard deviation over 5-fold cross-validation. Across all datasets, FairX consistently reduces GCIG relative to the unconstrained baseline and all outcome-focused baselines.

Citation

@misc{popoola2026procedural,
  title={Procedural Fairness via Group Counterfactual Explanation},
  author={Popoola et al. (2026)},
  year={2026},
  note={arXiv:2603.11140}
}

• arXiv: 2603.11140