Ood Detection 3d Medical Eval

ood-detection-3d-medical-eval

Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation — Vasiliuk et al. (2023) (arXiv:2306.13528, 2023)

What this evaluates

Evaluates the ability of out-of-distribution (OOD) detection methods to identify distribution shifts in 3D medical image segmentation. It measures how well models distinguish in-distribution scans from clinically anomalous or shifted OOD scans, highlighting the limitations of deep learning-based detectors compared to simpler intensity-based baselines.

Datasets

• 3D CT datasets — total ?; splits: train (-1), test (-1)
• 3D MRI datasets — total ?; splits: train (-1), test (-1)

Metrics

• FPR at 95% TPR (FPR95) (primary) — range: [0, 1]
  • False positive rate calculated at the operating point where the true positive rate is fixed at 95%. Lower values indicate better OOD detection performance, especially when outliers are rare.
• AUROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.

Input / output format

Input: 3D medical image volumes (CT or MRI) presented as patches of 64 axial slices, processed with minimum preprocessing required for 3D U-Net training.

Output: Binary segmentation mask per patch, plus an OOD detection score or decision for each sample used to compute FPR95 and AUROC.

Scoring recipe

def compute_fpr95(scores_id, scores_ood):
    tpr, fpr = compute_roc(scores_id, scores_ood)
    idx = np.where(tpr >= 0.95)[0][0]
    return fpr[idx]

Common pitfalls

• Relying solely on AUROC, which can be overly optimistic when OOD samples are rare compared to ID samples.
• Assuming deep learning-based OOD detectors will naturally outperform simple intensity-based baselines in medical imaging without explicit domain adaptation.

Evidence (verbatim from paper)

Given the testing part of the ID dataset, we measure the OOD detection quality against it for all the suggested OOD setups, similarly to the classification task. Outliers occur rarely in practice, so we aim to measure detection quality when most of the ID samples are being preserved w.r.t. relatively rare OOD events. In this case, one of the most convenient classification metrics to use is false positive rate at 95% true positive rate (FPR), so we consider FPR our primary metric Nonetheless, for the consistency with other studies, we report AUROC in the supplementary materials.

Citation

@misc{vasiliuk2023ood3dmedical,
  title={Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation},
  author={Vasiliuk et al. (2023)},
  year={2023},
  note={arXiv:2306.13528}
}

• arXiv: 2306.13528