Realistic Ood Detection Eval

realistic-ood-detection-eval

Towards Realistic Out-of-Distribution Detection: A Novel Evaluation Framework for Improving Generalization in OOD Detection — Khazaie et al. (2022) (arXiv:2211.10892, 2022)

What this evaluates

Evaluates the robustness of Out-of-Distribution (OOD) detection models under realistic distribution shifts caused by semantic-preserving transformations. It measures how well detectors distinguish between true out-of-distribution samples and inlier samples that have undergone common corruptions or augmentations, revealing performance gaps that standard benchmarks miss.

Datasets

• CIFAR-10-R — total 1001000; splits: test (1001000)
• CIFAR-100-R — total 1001000; splits: test (1001000)
• ImageNet-30-R — total 303000; splits: test (303000)

Metrics

• AUROC (primary) — range: [0, 100]
  • Area Under the Receiver Operating Characteristic Curve. Computed by plotting the True Positive Rate (inliers correctly identified) against the False Positive Rate (outliers misclassified as inliers) across all classification thresholds, and calculating the area under this curve. Reported as a percentage.
• Generalizability Score (GS) — range: other
  • A composite metric reported to quantify how well OOD detectors generalize under semantic-preserving distribution shifts. The exact mathematical formula is not provided in the experimental section.

Input / output format

Input: RGB images (32x32 for CIFAR variants, subset of ImageNet) passed through a pre-trained feature extractor (e.g., ViT, ResNet) to obtain feature vectors. A one-class OOD scoring function F_theta is applied to these features to produce a scalar anomaly score.

Output: A scalar OOD score per image. During evaluation, samples are binary-labeled: 1 for inliers (original class, its augmentations, and its corruptions), 0 for outliers (all other classes).

Scoring recipe

def compute_auroc(predictions, labels):
    # predictions: array of OOD scores (higher = more likely OOD)
    # labels: array of ground truth (1 for inlier, 0 for outlier)
    fpr, tpr, _ = roc_curve(labels, predictions)
    auroc = auc(fpr, tpr)
    return auroc * 100  # Reported as percentage

Common pitfalls

• Treating semantic-preserving transformations (augmentations/corruptions) of the inlier class as outliers; the protocol explicitly defines them as inliers.
• Assuming high AUROC on standard benchmarks guarantees real-world robustness; the framework demonstrates significant performance drops under realistic perturbations.
• Ignoring the one-class classification setup; only a single class is treated as inlier during both training and testing, while all other classes are outliers.

Evidence (verbatim from paper)

The results will be reported by averaging the AUROC of all classes. ... Provided in the table are the class-wise AUROC scores, the average AUROC scores over all classes, and the Generalizability Score (GS) before and after applying our proposed adaptation method to indicate the method’s applicability to real-world scenarios.

Citation

@misc{khazaie2022realistic,
  title={Towards Realistic Out-of-Distribution Detection: A Novel Evaluation Framework for Improving Generalization in OOD Detection},
  author={Khazaie et al. (2022)},
  year={2022},
  note={arXiv:2211.10892}
}

• arXiv: 2211.10892