Medical Ood Detection Eval

medical-ood-detection-eval

A Benchmark of Medical Out of Distribution Detection — Cao et al. (2020) (arXiv:2007.04250, 2020)

What this evaluates

Evaluates the ability of out-of-distribution detection (OoDD) methods to correctly classify medical images as in-distribution or out-of-distribution across three distinct OoD categories: unrelated domains, incorrect image preparation, and unseen medical conditions due to selection bias.

Datasets

• Medical OoD Benchmark ($D_{test}$) — total ?; splits: test (-1), val (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified In and Out samples on the class-balanced test set. Provides an unbiased representation of type I and type II errors.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall Curve. Characterizes the separability of In and Out samples based on predicted scores, which are thresholded to obtain binary classifications.

Input / output format

Input: Medical images (e.g., chest X-rays, fundus images, histology slides) processed by OoDD methods to generate In/Out scores or classifications.

Output: Binary classification predictions (In vs. Out) or continuous OoDD scores, thresholded to yield final predictions.

Scoring recipe

def compute_metrics(predictions, gold, scores):
    # predictions and gold are binary (0=In, 1=Out)
    accuracy = (predictions == gold).mean()
    # AUPRC requires continuous scores and binary gold labels
    precision, recall, _ = precision_recall_curve(gold, scores)
    auprc = auc(recall, precision)
    return accuracy, auprc

Common pitfalls

• Using a single Out dataset for validation ($D_{val}$) instead of randomly sampling multiple Out datasets, which reduces generalization and decision boundary stability.
• Assuming complex auxiliary models outperform simple classifier-only baselines; the benchmark shows simple methods often match or exceed complex ones.
• Underestimating the difficulty of Use-Case 3 (unseen diseases/selection bias), where methods perform near random chance.

Evidence (verbatim from paper)

We measure the accuracy and Area Under Precision-Recall Curve (AUPRC) on $D_{test}$ , totaling at 11 pairs of performance numbers per method. Since $D_{test}$ is class-balanced, accuracy provides an unbiased representation of type I and type II errors. AUPRC characterizes the separability of $In$ and $Out$ samples in predicted value (the value that we threshold to obtain classification).

Citation

@misc{cao2020benchmarkmedicalood,
  title={A Benchmark of Medical Out of Distribution Detection},
  author={Cao et al. (2020)},
  year={2020},
  note={arXiv:2007.04250}
}

• arXiv: 2007.04250