qhjqhj00 imagenet-o-eval
Natural Adversarial Examples — Hendrycks et al. (2019) (arXiv:1907.07174, 2019)
What this evaluates
Evaluates out-of-distribution (OOD) detection capabilities by measuring how well models assign low confidence to images of objects that do not exist in their training distribution. It probes whether models can reliably distinguish in-distribution classes from novel anomalies without relying on spurious cues.
Datasets
- ImageNet-O — total ?; splits: test (-1); repo https://github.com/hendrycks/natural-adv-examples
Metrics
AUPR(primary) — range: [0, 1]- Area under the precision-recall curve computed using anomaly scores defined as the negative of the maximum softmax probability across the 200 ImageNet-O classes.
Input / output format
Input: RGB images of out-of-distribution anomalies (e.g., triceratops, T-Rex) not present in the model's training classes.
Output: Maximum softmax probability across the 200 ImageNet-O classes (used to derive anomaly scores).
Scoring recipe
anomaly_scores = [-max(softmax_probs) for probs in model_outputs]
precision, recall, _ = precision_recall_curve(gold_labels, anomaly_scores)
aupr = auc(recall, precision)
Common pitfalls
- Random chance AUPR is ~16.67%, so scores must be interpreted relative to this baseline rather than assuming higher is always linearly better.
- ImageNet-O images may overlap with ImageNet-21K training data, which can artificially inflate OOD detection performance if not accounted for.
Evidence (verbatim from paper)
Our metric for assessing out-of-distribution detection performance of ImageNet-O examples is the area under the precision-recall curve (AUPR). This metric requires anomaly scores. Our anomaly score is the negative of the maximum softmax probabilities from a model that can classify the 200 ImageNet-O classes.
Citation
@misc{hendrycks2019naturaladversarialexamples,
title={Natural Adversarial Examples},
author={Hendrycks et al. (2019)},
year={2019},
note={arXiv:1907.07174}
}
- arXiv: 1907.07174