Relaxed Equal Odds Fraud Eval

relaxed-equal-odds-fraud-eval

Multiple Attribute Fairness: Application to Fraud Detection — Meghanath M Y et al. (2022) (arXiv:2207.14355, 2022)

What this evaluates

Evaluates a model-agnostic fairness calibration heuristic that adjusts prediction thresholds per protected attribute value to independently control false positive and false negative rates. It probes the ability to balance business-critical error costs across high-arity and multiple sensitive groups while maintaining predictive performance.

Datasets

• COMPAS (Criminal Recidivism) — total ?; splits: test (-1); repo https://github.com/propublica/compas-analysis
• Income-Prediction — total ?; splits: test (-1)
• Health Prediction — total ?; splits: test (-1); repo https://github.com/gpleiss/equalized_odds_and_calibration/
• Proprietary Online Fraud Dataset — total ?; splits: test (-1)

Metrics

• FPR (primary) — range: [0, 1]
  • False Positive Rate = FP / (FP + TN). Computed independently per protected attribute group to measure the rate of non-fraudulent transactions incorrectly flagged as fraudulent (or vice versa depending on class labeling).
• FNR — range: [0, 1]
  • False Negative Rate = FN / (FN + TP). Computed independently per protected attribute group to measure the rate of fraudulent transactions incorrectly flagged as non-fraudulent.

Input / output format

Input: Model prediction scores/probabilities and ground truth labels, along with protected attribute values (e.g., country, currency, gender, race) for post-hoc threshold adjustment.

Output: Adjusted binary predictions or custom thresholds per protected attribute value that satisfy relaxed equalized odds constraints.

Scoring recipe

def compute_fpr_fnr(preds, labels, groups):
    results = {}
    for g in groups.unique():
        mask = groups == g
        fp = ((preds[mask] == 1) & (labels[mask] == 0)).sum()
        tn = ((preds[mask] == 0) & (labels[mask] == 0)).sum()
        fn = ((preds[mask] == 0) & (labels[mask] == 1)).sum()
        tp = ((preds[mask] == 1) & (labels[mask] == 1)).sum()
        results[g] = {
            'FPR': fp / (fp + tn) if (fp + tn) > 0 else 0.0,
            'FNR': fn / (fn + tp) if (fn + tp) > 0 else 0.0
        }
    return results

Common pitfalls

• Confusing the proposed post-hoc threshold adjustment with pre-processing or in-processing fairness methods.
• Assuming standard equalized odds requires equal FPR and FNR simultaneously; the paper explicitly decouples them into 'relaxed equalized odds' to allow independent control.
• Reporting aggregate FPR/FNR instead of per-group values, which masks the fairness violations the method aims to fix.

Evidence (verbatim from paper)

From Figures 3, 4 (green color bars), we observe that the final $F_{select}$ with custom thresholds per country considerably reduces the bias towards certain countries. Further, we also note that the FPRs and FNRs across countries conform to the relaxed equalized odds fairness measure – mean across the countries lies withing two standard deviations.

Citation

@misc{meghanath2022multiple,
  title={Multiple Attribute Fairness: Application to Fraud Detection},
  author={Meghanath M Y et al. (2022)},
  year={2022},
  note={arXiv:2207.14355}
}

• arXiv: 2207.14355