# Caad 3k Eval

> Evaluates a model's ability to detect contextual anomalies where normality depends on subject-context alignment rather than intrinsic appearance. It probes cross-context generalization by testing on unseen subject-context combinations and zero-shot transfer to real-world out-of-context benchmarks. Use when the user wants to benchmark on CAAD-3K, MVTec-AD, VisA, MIT-OOC, COCO-OOC, or asks about evaluating this task. Reports I-AUROC.

- Skill: `qhjqhj00/caad-3k-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/caad-3k-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/caad-3k-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/caad-3k-eval

---


# caad-3k-eval

> When Anomalies Depend on Context: Learning Conditional Compatibility for Anomaly Detection — Mishra et al. (2026) (arXiv:2601.22868, 2026)

## What this evaluates

Evaluates a model's ability to detect contextual anomalies where normality depends on subject-context alignment rather than intrinsic appearance. It probes cross-context generalization by testing on unseen subject-context combinations and zero-shot transfer to real-world out-of-context benchmarks.

## Datasets

- **CAAD-3K** — total 3000; splits: CAAD-SS (2095), CAAD-CC (905)
- **MVTec-AD** — total ?; splits: test (-1)
- **VisA** — total ?; splits: test (-1)
- **MIT-OOC** — total ?; splits: test (-1)
- **COCO-OOC** — total ?; splits: test (-1)

## Metrics

- `I-AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve computed at the image level to measure anomaly detection performance.
- `P-AUROC` — range: [0, 1]
  - Pixel-level AUROC used to evaluate localization quality on datasets with pixel-level annotations.

## Input / output format

**Input**: Single RGB image resized to a fixed resolution. For few-shot evaluation, a small support set of labeled images is provided. For zero-shot transfer, only class names are provided via text prompts at inference.

**Output**: Anomaly score per image (and per pixel for localization). Performance is aggregated as AUROC across the dataset.

## Scoring recipe

```python
def compute_i_auroc(predictions, labels):
    # predictions: anomaly scores per image
    # labels: 1 for anomaly, 0 for normal
    fpr, tpr, _ = roc_curve(labels, predictions)
    return auc(fpr, tpr)
```

## Common pitfalls

- Confusing contextual anomalies (subject-context mismatch) with structural/appearance anomalies (defects in texture or shape).
- Assuming segmentation masks are required at inference; they are only used during training for region decomposition.
- Evaluating cross-context generalization without holding out unseen subject-context combinations at test time.

## Evidence (verbatim from paper)

> We report Area Under the ROC Curve (AUROC) as the primary evaluation metric. Image-level AUROC (I-AUROC) is used to measure anomaly detection performance and serves as the main metric for CAAD-3K, as the task focuses on identifying subject–context incompatibility.

## Citation

```bibtex
@misc{mishra2026caad,
  title={When Anomalies Depend on Context: Learning Conditional Compatibility for Anomaly Detection},
  author={Mishra et al. (2026)},
  year={2026},
  note={arXiv:2601.22868}
}
```

- arXiv: 2601.22868