Generalad Eval

generalad-eval

GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features — Sträter et al. (2024) (arXiv:2407.12427, 2024)

What this evaluates

Evaluates cross-domain anomaly detection capability across semantic, near-distribution, and industrial benchmarks. Probes the model's ability to detect pixel-level defects and semantic novelties using a self-supervised transformer discriminator that attends to distorted features without task-specific tuning.

Datasets

• CIFAR-10 — total ?; splits: train (-1), test (-1)
• CIFAR-100 — total ?; splits: train (-1), test (-1)
• Fashion-MNIST — total ?; splits: train (-1), test (-1)
• View — total ?; splits: train (-1), test (-1)
• Aircraft-FGVC — total ?; splits: train (-1), test (-1)
• Stanford Cars — total ?; splits: train (-1), test (-1)
• MVTec-AD — total ?; splits: train (-1), test (-1)
• MVTec-LOCO — total ?; splits: train (-1), test (-1)
• VisA — total ?; splits: train (-1), test (-1)
• MPDD — total ?; splits: train (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Computed separately for image-level binary classification and pixel-level segmentation/localization tasks.

Input / output format

Input: Input images rescaled to 518×518 without augmentations. Processed through a DINOv2 backbone to extract last-layer features, which are normalized. For training, features are distorted via noise injection and patch shuffling to create pseudo-abnormal samples.

Output: Anomaly scores at the image level and pixel-level anomaly score maps. Thresholded to produce binary anomaly labels or segmentation masks.

Scoring recipe

def compute_auroc(scores, labels):
    # scores: anomaly scores (higher = more anomalous)
    # labels: ground truth (0 = normal, 1 = anomalous)
    fpr, tpr, _ = roc_curve(labels, scores)
    return auc(fpr, tpr)

Common pitfalls

• The top-K parameter for feature selection must be dataset-specific: K=1369 for semantic/near datasets (covering all patches) and K=10 for industrial datasets (covering small defects).
• Distortion strategies are task-dependent: 'Noise All Patches' for semantic, 'Noise Random Patches' for industrial, and 'Attention Shuffle' for logical anomalies.
• Input images must be rescaled to 518×518 without any image augmentations to maintain consistency with the DINOv2 backbone.

Evidence (verbatim from paper)

We report the AUROC performance for both image-level and pixel-level tasks in tables[1] and[2]. Our experiments encompass three distinct benchmarks. First, we evaluate our approach on four semantic anomaly detection datasets: CIFAR-10, CIFAR-100, Fashion-MNIST, and View. ... Finally, we experiment on four industrial anomaly detection datasets: MVTec-AD, MVTec-LOCO, VisA, and MPDD.

Citation

@misc{strater2024generalad,
  title={GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features},
  author={Sträter et al. (2024)},
  year={2024},
  note={arXiv:2407.12427}
}

• arXiv: 2407.12427