Ood Detection Eval

ood-detection-eval

A Novel Data Augmentation Technique for Out-of-Distribution Sample Detection using Compounded Corruptions — Hebbalaguppe et al. (2022) (arXiv:2207.13916, 2022)

What this evaluates

Evaluates a model's ability to distinguish between in-distribution (ID) samples and out-of-distribution (OOD) samples. It measures how well the model's decision boundaries separate known classes from unknown data distributions using various synthetic and natural OOD benchmarks.

Datasets

• SVHN — total ?; splits: train (-1), test (-1)
• CIFAR-10 — total ?; splits: train (-1), test (-1)
• CIFAR-100 — total ?; splits: train (-1), test (-1)
• TinyImageNet — total ?; splits: train (-1), test (-1)
• TinyImageNet-crop — total ?; splits: test (-1)
• TinyImageNet-resize — total ?; splits: test (-1)
• LSUN-crop — total ?; splits: test (-1)
• LSUN-resize — total ?; splits: test (-1)
• iSUN — total ?; splits: test (-1)

Metrics

• TNR@TPR95 (primary) — range: percent
  • True Negative Rate at a fixed True Positive Rate of 95%. Computed from the ROC curve of OOD scores on ID vs OOD samples.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the trade-off between true positive and false positive rates across all thresholds.
• Detection Error — range: percent
  • Classification error rate for OOD detection, typically calculated as the average of false positive and false negative rates at the optimal threshold, or 1 - AUROC.
• ID Acc — range: percent
  • Standard top-1 classification accuracy on the in-distribution test set.

Input / output format

Input: RGB images (32x32 or 64x64 pixels) fed into a trained neural network classifier to extract logits or softmax probabilities.

Output: Per-class probability scores or logits. OOD scores are derived (e.g., 1 - max softmax probability), and metrics are computed per dataset and averaged across all OOD benchmarks.

Scoring recipe

# 1. Compute OOD score for each sample (e.g., 1 - max_softmax(logits))
# 2. Split scores into ID (label=0) and OOD (label=1) arrays
# 3. Compute ROC curve: fpr, tpr, _ = roc_curve(y_true, ood_scores)
# 4. TNR@TPR95: Find FPR where TPR >= 0.95, convert to percentage
# 5. AUROC: Integrate ROC curve (trapz)
# 6. DetErr: (FPR + FNR) / 2 at operating point, or 1 - AUROC
# 7. Average each metric across all OOD datasets
# 8. Report mean ± std over 3 independent runs

Common pitfalls

• Main table reports metrics averaged across all OOD benchmark datasets, not per-dataset values.
• CnC results include standard deviation over 3 runs, while baseline methods often report single-point values.
• OOD datasets include both natural images (LSUN, iSUN) and processed TinyImageNet variants, requiring consistent preprocessing pipelines.

Evidence (verbatim from paper)

Evaluation Metrics: We compare the performance of various approaches using TNR@TPR95, AUROC and Detection Error. See Suppl. for description on evaluation metrics. Values are averaged over all OOD benchmark datasets. CnC reasults are averaged on 3 evaluation runs.

Citation

@misc{hebbalaguppe2022cnc,
  title={A Novel Data Augmentation Technique for Out-of-Distribution Sample Detection using Compounded Corruptions},
  author={Hebbalaguppe et al. (2022)},
  year={2022},
  note={arXiv:2207.13916}
}

• arXiv: 2207.13916