Ood Detection Cxr Eval

ood-detection-cxr-eval

A knee cannot have lung disease: out-of-distribution detection with in-distribution voting using the medical example of chest X-ray classification — Wollek et al. (2022) (arXiv:2208.01077, 2022)

What this evaluates

Evaluates a model's ability to distinguish in-distribution chest X-rays from out-of-distribution medical images (e.g., knee, hand, or general radiographs) while maintaining classification accuracy on chest diseases.

Datasets

• CXR14 — total ?; splits: test (-1)
• IRMA — total ?; splits: test (-1)
• MURA — total ?; splits: test (-1)
• Bone Age — total ?; splits: test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen in-distribution sample is ranked higher than a randomly chosen out-of-distribution sample by the model's OOD detection score.

Input / output format

Input: Chest X-ray images (radiographs) from various anatomical regions or general datasets.

Output: Binary OOD detection score (probability of being in-distribution) or 15-class disease probabilities for ID classification.

Scoring recipe

def compute_auc(scores, labels):
    fpr, tpr, _ = roc_curve(labels, scores)
    return auc(fpr, tpr)

# For each OOD test dataset:
for dataset in [IRMA, MURA, BoneAge]:
    scores = model.get_ood_scores(dataset.images)
    labels = [1] * len(dataset.id_samples) + [0] * len(dataset.ood_samples)
    dataset_aucs.append(compute_auc(scores, labels))
mean_auc = sum(dataset_aucs) / len(dataset_aucs)

Common pitfalls

• Training with large OOD datasets (e.g., full ImageNet) can degrade in-distribution classification AUC by up to 3 percentage points.
• Self-supervised auxiliary heads (SS OOD) negatively impact both OOD detection and ID classification performance.
• Domain-specific OOD data (IRMA) generalizes better to X-ray OODs than general datasets (ImageNet), but combining both yields the highest AUC.

Evidence (verbatim from paper)

The objective of OOD detection is to classify each image as either ID or OOD. For each of the three OOD data sets (IRMA, MURA, and BoneAge), we evaluate the performance of the OOD detection methods by measuring how many ID and OOD samples from the test set are correctly classified as such. As a baseline, we employed the default CheXnet model with no extra OOD detection mechanism, which represents the current CXR classification models. We report the AUC as our evaluation metric.

Citation

@misc{wollek2022ooddetection,
  title={A knee cannot have lung disease: out-of-distribution detection with in-distribution voting using the medical example of chest X-ray classification},
  author={Wollek et al. (2022)},
  year={2022},
  note={arXiv:2208.01077}
}

• arXiv: 2208.01077