Compas Fairness Eval

compas-fairness-eval

FADE: FAir Double Ensemble Learning for Observable and Counterfactual Outcomes — Mishler et al. (2021) (arXiv:2109.00173, 2021)

What this evaluates

Evaluates a fairness-aware ensemble learning framework on recidivism risk prediction. It probes the model's ability to balance predictive accuracy against multiple group fairness constraints across racial demographics in a counterfactual causal setting.

Datasets

• COMPAS — total ?; splits: D_learn (-1), D_train_nuis (-1), D_train_target (-1), D_test_nuis (-1), D_test_target (-1)

Metrics

• MSE (primary) — range: [0, 1]
  • Mean squared error between predicted risk scores and actual rearrest outcomes.
• rate-diff — range: [0, 1]
  • Absolute difference in predicted positive rates between African-American and Caucasian defendants.
• FPR-diff — range: [0, 1]
  • Absolute difference in false positive rates (P(pred=1|y=0)) between racial groups.
• FNR-diff — range: [0, 1]
  • Absolute difference in false negative rates (P(pred=0|y=1)) between racial groups.

Input / output format

Input: Three features: indicator for age > 45, indicator for age < 25, and number of prior arrests (0-29). Sensitive attribute: race (African-American or Caucasian).

Output: Continuous risk score normalized to the range [0.1, 1].

Scoring recipe

def compute_metrics(y_true, y_pred, sensitive):
    mse = np.mean((y_pred - y_true) ** 2)
    # Threshold t required for FPR/FNR; not specified in text
    y_pred_bin = (y_pred >= t).astype(int)
    y_true_bin = (y_true >= t).astype(int)
    groups = ['AA', 'C']
    rates = {g: np.mean(y_pred_bin[sensitive==g]) for g in groups}
    fprs = {g: np.mean(y_pred_bin[sensitive==g] & ~y_true_bin[sensitive==g]) for g in groups}
    fnrs = {g: np.mean(~y_pred_bin[sensitive==g] & y_true_bin[sensitive==g]) for g in groups}
    return {
        'MSE': mse,
        'rate-diff': abs(rates['AA'] - rates['C']),
        'FPR-diff': abs(fprs['AA'] - fprs['C']),
        'FNR-diff': abs(fnrs['AA'] - fnrs['C'])
    }

Common pitfalls

• The paper does not specify the decision threshold used to compute FPR and FNR from continuous risk scores.
• The evaluation uses a non-standard 5-way data split for doubly robust estimation, which differs from standard train/test splits.
• Disparities are reported as absolute differences between two racial groups, but the exact binarization of the continuous COMPAS scores for rate/FPR/FNR is not explicitly defined.

Evidence (verbatim from paper)

We split the data into five datasets, each with approximately 1040 rows: D_learn, D_train_nuis, D_train_target, D_test_nuis, and D_test_target. As base predictors, we used the four model types from the previous section as well as a logistic regression. We used random forest classifiers for the nuisance predictors in both the training and test data. Table 7 gives the estimated performance of the five base predictors, COMPAS, and the OLS predictor, which spans COMPAS and the base predictors. ... We compute FADE predictors using the same sets of penalty vectors Λ as in the previous section. Figure 7 shows disparities and MSE values for all 1331 predictors.

Citation

@misc{mishler2021fade,
  title={FADE: FAir Double Ensemble Learning for Observable and Counterfactual Outcomes},
  author={Mishler et al. (2021)},
  year={2021},
  note={arXiv:2109.00173}
}

• arXiv: 2109.00173