Caad 3k Eval

caad-3k-eval

When Anomalies Depend on Context: Learning Conditional Compatibility for Anomaly Detection — Mishra et al. (2026) (arXiv:2601.22868, 2026)

What this evaluates

Evaluates a model's ability to detect contextual anomalies where normality depends on subject-context alignment rather than intrinsic appearance. It probes cross-context generalization by testing on unseen subject-context combinations and zero-shot transfer to real-world out-of-context benchmarks.

Datasets

• CAAD-3K — total 3000; splits: CAAD-SS (2095), CAAD-CC (905)
• MVTec-AD — total ?; splits: test (-1)
• VisA — total ?; splits: test (-1)
• MIT-OOC — total ?; splits: test (-1)
• COCO-OOC — total ?; splits: test (-1)

Metrics

• I-AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve computed at the image level to measure anomaly detection performance.
• P-AUROC — range: [0, 1]
  • Pixel-level AUROC used to evaluate localization quality on datasets with pixel-level annotations.

Input / output format

Input: Single RGB image resized to a fixed resolution. For few-shot evaluation, a small support set of labeled images is provided. For zero-shot transfer, only class names are provided via text prompts at inference.

Output: Anomaly score per image (and per pixel for localization). Performance is aggregated as AUROC across the dataset.

Scoring recipe

def compute_i_auroc(predictions, labels):
    # predictions: anomaly scores per image
    # labels: 1 for anomaly, 0 for normal
    fpr, tpr, _ = roc_curve(labels, predictions)
    return auc(fpr, tpr)

Common pitfalls

• Confusing contextual anomalies (subject-context mismatch) with structural/appearance anomalies (defects in texture or shape).
• Assuming segmentation masks are required at inference; they are only used during training for region decomposition.
• Evaluating cross-context generalization without holding out unseen subject-context combinations at test time.

Evidence (verbatim from paper)

We report Area Under the ROC Curve (AUROC) as the primary evaluation metric. Image-level AUROC (I-AUROC) is used to measure anomaly detection performance and serves as the main metric for CAAD-3K, as the task focuses on identifying subject–context incompatibility.

Citation

@misc{mishra2026caad,
  title={When Anomalies Depend on Context: Learning Conditional Compatibility for Anomaly Detection},
  author={Mishra et al. (2026)},
  year={2026},
  note={arXiv:2601.22868}
}

• arXiv: 2601.22868