# Imagenet O Eval

> Evaluates out-of-distribution (OOD) detection capabilities by measuring how well models assign low confidence to images of objects that do not exist in their training distribution. It probes whether models can reliably distinguish in-distribution classes from novel anomalies without relying on spurious cues. Use when the user wants to benchmark on ImageNet-O, or asks about evaluating this task. Reports AUPR.

- Skill: `qhjqhj00/imagenet-o-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/imagenet-o-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/imagenet-o-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/imagenet-o-eval

---


# imagenet-o-eval

> Natural Adversarial Examples — Hendrycks et al. (2019) (arXiv:1907.07174, 2019)

## What this evaluates

Evaluates out-of-distribution (OOD) detection capabilities by measuring how well models assign low confidence to images of objects that do not exist in their training distribution. It probes whether models can reliably distinguish in-distribution classes from novel anomalies without relying on spurious cues.

## Datasets

- **ImageNet-O** — total ?; splits: test (-1); repo https://github.com/hendrycks/natural-adv-examples

## Metrics

- `AUPR` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve computed using anomaly scores defined as the negative of the maximum softmax probability across the 200 ImageNet-O classes.

## Input / output format

**Input**: RGB images of out-of-distribution anomalies (e.g., triceratops, T-Rex) not present in the model's training classes.

**Output**: Maximum softmax probability across the 200 ImageNet-O classes (used to derive anomaly scores).

## Scoring recipe

```python
anomaly_scores = [-max(softmax_probs) for probs in model_outputs]
precision, recall, _ = precision_recall_curve(gold_labels, anomaly_scores)
aupr = auc(recall, precision)
```

## Common pitfalls

- Random chance AUPR is ~16.67%, so scores must be interpreted relative to this baseline rather than assuming higher is always linearly better.
- ImageNet-O images may overlap with ImageNet-21K training data, which can artificially inflate OOD detection performance if not accounted for.

## Evidence (verbatim from paper)

> Our metric for assessing out-of-distribution detection performance of ImageNet-O examples is the area under the precision-recall curve (AUPR). This metric requires anomaly scores. Our anomaly score is the negative of the maximum softmax probabilities from a model that can classify the 200 ImageNet-O classes.

## Citation

```bibtex
@misc{hendrycks2019naturaladversarialexamples,
  title={Natural Adversarial Examples},
  author={Hendrycks et al. (2019)},
  year={2019},
  note={arXiv:1907.07174}
}
```

- arXiv: 1907.07174