Medfair Eval

medfair-eval

MEDFAIR: Benchmarking Fairness for Medical Imaging — Zong et al. (2022) (arXiv:2210.01725, 2022)

What this evaluates

Evaluates group fairness and bias mitigation in medical imaging models across multiple datasets and modalities. It probes whether models trained with Empirical Risk Minimization (ERM) or explicit bias mitigation algorithms exhibit performance disparities across sensitive subgroups, and how model selection strategies impact worst-case group performance.

Datasets

• MEDFAIR Benchmark Suite — total ?; splits: train (-1), test (-1); repo https://github.com/ys-zong/MEDFAIR

Metrics

• worst-case AUC (primary) — range: [0, 1]
  • The Area Under the Receiver Operating Characteristic Curve computed on the sensitive subgroup with the lowest performance. It serves as the headline fairness metric to ensure no group is left behind.
• AUC gap — range: [0, 1]
  • The difference between the maximum and minimum AUC across all sensitive subgroups (max(AUC) - min(AUC)). A smaller gap indicates better group fairness.
• overall AUC — range: [0, 1]
  • The standard Area Under the ROC Curve computed across the entire dataset without subgroup stratification, used as a baseline for overall diagnostic performance.
• underdiagnosis rate — range: [0, 1]
  • Defined as the False Negative Rate (FNR) for malignant labels, or the False Positive Rate (FPR) for 'No Finding' labels, calculated per subgroup to capture clinical error direction.

Input / output format

Input: Medical images paired with diagnostic labels and sensitive demographic attributes (e.g., age, sex, race).

Output: Continuous prediction scores or probabilities for the diagnostic task, aggregated per sensitive subgroup for metric computation.

Scoring recipe

def compute_metrics(preds, labels, subgroups):
    subgroup_aucs = [auc_score(preds[s], labels[s]) for s in subgroups]
    worst_auc = min(subgroup_aucs)
    auc_gap = max(subgroup_aucs) - min(subgroup_aucs)
    overall_auc = auc_score(preds, labels)
    underdiag = fnr(preds, labels) if label=='malignant' else fpr(preds, labels)
    return worst_auc, auc_gap, overall_auc, underdiag

Common pitfalls

• Using standard overall-AUC model selection instead of worst-case or Pareto selection, which masks severe subgroup disparities and leads to poor worst-group performance.
• Misinterpreting underdiagnosis rate: FNR is used for malignant labels, but FPR is used for 'No Finding' labels, reversing the typical error direction depending on the task.
• Drawing conclusions from insufficient dataset counts; the paper emphasizes using Nemenyi post-hoc tests across many datasets to establish statistical significance for fairness claims.

Evidence (verbatim from paper)

For each dataset and sensitive attribute, we calculate the maximum and minimum AUC and underdiagnosis rate among subgroups, where we use FNR for the malignant label and FPR for "No Finding" label as the underdiagnosis rate.

Citation

@misc{zong2022medfair,
  title={MEDFAIR: Benchmarking Fairness for Medical Imaging},
  author={Zong et al. (2022)},
  year={2022},
  note={arXiv:2210.01725}
}

• arXiv: 2210.01725