visA-mvtec-ad-eval

visA-mvtec-ad-eval

SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation — Yang Zou et al. (2022) (arXiv:2207.14315, 2022)

What this evaluates

Evaluates industrial anomaly detection and segmentation capabilities by measuring how well self-supervised pre-training methods transfer to identifying surface defects. It probes a model's ability to localize fine-grained anomalies in highly imbalanced, high-resolution industrial imagery under both one-class and few-shot supervised regimes.

Datasets

• VisA — total 10821; splits: test (-1)
• MVTec-AD — total 5354; splits: test (-1)

Metrics

• AU-PR (primary) — range: [0, 1]
  • Area Under the Precision-Recall curve. Computed by integrating precision over recall thresholds for predicted anomaly scores against ground truth labels or masks.
• AU-ROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Computed by integrating the True Positive Rate over the False Positive Rate thresholds for predicted anomaly scores.

Input / output format

Input: High-resolution industrial images (normal and anomalous) used for pre-training and downstream anomaly detection/segmentation tasks.

Output: Per-image anomaly score (classification) and per-pixel binary anomaly mask (segmentation).

Scoring recipe

def compute_aupr(y_true, y_pred):
    precisions, recalls, _ = precision_recall_curve(y_true, y_pred)
    return auc(recalls, precisions)

def compute_auroc(y_true, y_pred):
    fprs, tprs, _ = roc_curve(y_true, y_pred)
    return auc(fprs, tprs)

Common pitfalls

• Relying on AU-ROC as the sole metric, which saturates near 1.0 and masks poor performance on highly imbalanced industrial anomaly data.
• Confusing the 1-class (unsupervised/one-class) and 2-class (supervised) evaluation regimes, which exhibit significantly different performance gaps and baseline rankings.
• Overlooking the distinction between high-shot (full anomaly set) and low-shot (5/10 anomaly samples) training conditions, which drastically changes model behavior and metric trends.

Evidence (verbatim from paper)

While we report both AU-PR and AU-ROC, the former metric is more relevant to the application and we see that self-supervised methods are improved up to AU-PR of $2.6%$. ... Moreover, the gap between low AU-PR and high AU-ROC for both VisA/MVTec segmentation justifies the inflated performance view of AU-ROC, in favor of AU-PR as a more suitable metric in imbalanced datasets.

Citation

@misc{zou2022spot,
  title={SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation},
  author={Yang Zou et al. (2022)},
  year={2022},
  note={arXiv:2207.14315}
}

• arXiv: 2207.14315