Mimic Iii Clinical Fairness Eval

mimic-iii-clinical-fairness-eval

Exploring Text Specific and Blackbox Fairness Algorithms in Multimodal Clinical NLP — Chen et al. (2020) (arXiv:2011.09625, 2020)

What this evaluates

Probes multimodal clinical NLP models on in-hospital mortality and phenotyping tasks, evaluating both predictive performance (AUC) and group fairness (Equalized Odds) across protected demographic groups.

Datasets

• MIMIC-III — total 60000; splits: train (-1), test (-1); repo https://github.com/johntiger1/multimodal_fairness

Metrics

• AUC ROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• AUC PRC — range: [0, 1]
  • Area under the Precision-Recall Curve, emphasizing performance on the positive class in highly imbalanced datasets.
• Equalized Odds (EO) — range: [0, 1]
  • The absolute difference in True Positive Rates (TPR) and False Positive Rates (FPR) between protected and unprotected groups; lower difference indicates better fairness.

Input / output format

Input: Multimodal data per ICU admission: structured time-series/demographic features and unstructured clinical notes linked to the stay.

Output: Binary classification probabilities (e.g., mortality risk or phenotyping label) used to compute AUC scores and group-wise TPR/FPR for fairness assessment.

Scoring recipe

preds: predicted probabilities, y_true: ground truth labels, groups: protected group IDs
auc_roc = roc_auc_score(y_true, preds)
auc_prc = average_precision_score(y_true, preds)
tpr_diff = abs(mean(preds[(y_true==1) & (groups==1)]) - mean(preds[(y_true==1) & (groups==0)]))
fpr_diff = abs(mean(preds[(y_true==0) & (groups==1)]) - mean(preds[(y_true==0) & (groups==0)]))
eo_gap = max(tpr_diff, fpr_diff)

Common pitfalls

• Severe class imbalance (e.g., 13.1% positive for IHM) makes standard accuracy misleading; AUC PRC is essential for reliable evaluation.
• Naively enforcing Equalized Odds can cause unacceptable performance drops; clinical deployment requires stakeholder consultation beyond blind metric optimization.

Evidence (verbatim from paper)

To account for the label imbalance we evaluate performance using AUC ROC and AUC PRC. ... Attempting to equalize these rates corresponds to satisfying Equalized Odds. EO satisfies many desiderata within clinical settings, and has been used in previous clinical fairness work (Pfohl et al., 2019a; Garb, 1997; Pfohl et al., 2019b).

Citation

@misc{chen2020multimodal,
  title={Exploring Text Specific and Blackbox Fairness Algorithms in Multimodal Clinical NLP},
  author={Chen et al. (2020)},
  year={2020},
  note={arXiv:2011.09625}
}

• arXiv: 2011.09625