Fair Inference Causal Mediation Eval

fair-inference-causal-mediation-eval

Fair Inference On Outcomes — Nabi et al. (2017) (arXiv:1705.10378, 2017)

What this evaluates

Evaluates whether a predictive model can satisfy fairness constraints defined by causal mediation analysis (NDE/PSE) while maintaining out-of-sample accuracy. It probes the model's ability to isolate and eliminate discriminatory pathways from sensitive attributes to outcomes without relying on fully specified outcome models.

Datasets

• COMPAS — total 11000; splits: train (-1), test (-1)
• Adult (UCI) — total 48842; splits: train (-1), test (-1)

Metrics

• NDE (odds ratio) (primary) — range: odds ratio
  • Natural Direct Effect computed as the odds ratio of the counterfactual outcome when the sensitive attribute is changed, holding mediators fixed to their natural values under the original treatment. Calculated via the functional E[Y(a, M(a), C)] / E[Y(a', M(a'), C)].
• Test accuracy — range: percent
  • Standard classification accuracy calculated as the proportion of correct predictions on the held-out test set.

Input / output format

Input: Tabular records containing sensitive attribute (A), mediators (M), confounders (C), and binary outcome (Y).

Output: Predicted outcome class/probability, estimated NDE/PSE value, and model parameters constrained to satisfy the fairness bound.

Scoring recipe

# 1. Estimate counterfactual distributions using BART or regression
p_Y_given_AMC = model.predict(Y | A, M, C)
p_M_given_AC = model.predict(M | A, C)
# 2. Compute NDE/PSE via path-specific functional
nde = compute_odds_ratio(
    E[Y(a=1, M(a=1), C)], 
    E[Y(a=0, M(a=0), C)]
)
# 3. Enforce fairness constraint during optimization
if not (0.95 <= nde <= 1.05):
    adjust_model_parameters()
# 4. Evaluate predictive performance
accuracy = mean(y_pred == y_true)

Common pitfalls

• Dropping the sensitive feature from the model does not remove discrimination as defined by NDE/PSE, because the effect still flows through mediators.
• Selecting outcome models solely to maximize prediction accuracy can yield biased NDE/PSE estimates unless robust estimators (e.g., IPW, triply robust) are used to handle potential misspecification.

Evidence (verbatim from paper)

Under the unconstrained posterior, the NDE (on the odds ratio scale) was equal to 1.3. This number is interpreted to mean that the odds of recidivism would have been 1.3 times higher had we changed race from Caucasian to African-American. In our experiment we restricted NDE to lie between 0.95 and 1.05. Using unconstrained BART, our prediction accuracy on the test set was 67.8%, removing treatment from the outcome model dropped the accuracy to 64.0%, and using constrained BART lead to the accuracy of 66.4%.

Citation

@misc{nabi2017fairinference,
  title={Fair Inference On Outcomes},
  author={Nabi et al. (2017)},
  year={2017},
  note={arXiv:1705.10378}
}

• arXiv: 1705.10378