Mvtec Ad Eval

mvtec-ad-eval

Towards Total Recall in Industrial Anomaly Detection — Roth et al. (2021) (arXiv:2106.08265, 2021)

What this evaluates

Evaluates an image classification and segmentation model's ability to detect and localize defects in industrial products without seeing anomalous examples during training. It probes the model's capacity to learn nominal feature distributions and identify deviations at both image and pixel levels.

Datasets

• MVTec AD — total 5354; splits: train (-1), test (1725)
• Magnetic Tile Defects (MTD) — total 1317; splits: train (-1), test (-1)
• Mini Shanghai Tech Campus (mSTC) — total ?; splits: train (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve computed over image-level anomaly scores. On MVTec AD, the class-average AUROC is reported.
• Pixel-wise AUROC — range: [0, 1]
  • Area under the ROC curve computed over pixel-level anomaly scores for segmentation masks.
• PRO — range: [0, 1]
  • Measures the overlap and recovery of connected anomaly components to better account for varying anomaly sizes in industrial datasets.
• Error — range: percent
  • Sum of false positives and false negatives at the F1-optimal threshold.

Input / output format

Input: RGB images resized and center-cropped to 256×256 or 224×224. No data augmentation is applied. Training data contains only nominal (defect-free) images.

Output: Per-image anomaly score for classification. Per-pixel anomaly map/mask for segmentation.

Scoring recipe

def evaluate(scores, labels, masks=None, gt_masks=None):
    class_auroc = np.mean([roc_auc_score(labels[c], scores[c]) for c in np.unique(labels)])
    best_f1, best_err = 0, float('inf')
    for t in np.unique(scores):
        preds = (scores >= t).astype(int)
        fp, fn = np.sum((preds==1)&(labels==0)), np.sum((preds==0)&(labels==1))
        err = fp + fn
        f1 = 2 * (1 - err/len(labels)) if len(labels) > 0 else 0
        if f1 > best_f1: best_f1, best_err = f1, err
    error_pct = (best_err / len(labels)) * 100
    pw_auroc = roc_auc_score(gt_masks.flatten(), masks.flatten())
    pro = compute_pro(gt_masks, masks)
    return class_auroc, pw_auroc, pro, error_pct

Common pitfalls

• MVTec AD requires class-average AUROC, not image-average.
• PRO metric evaluates connected component overlap/recovery, not simple pixel-wise IoU.
• Error rate is computed at the F1-optimal threshold, which must be searched per dataset/method.
• Training uses only nominal data; anomalous images are strictly for testing.

Evidence (verbatim from paper)

Image-level anomaly detection performance is measured via the area under the receiver-operator curve (AUROC) using produced anomaly scores. In accordance with prior work we compute on MVTec the class-average AUROC. To measure segmentation performance, we use both pixel-wise AUROC and the PRO metric first, both following[6]. The PRO score takes into account the overlap and recovery of connected anomaly components to better account for varying anomaly sizes in MVTec AD, see[6] for details.

Citation

@misc{roth2021totalrecall,
  title={Towards Total Recall in Industrial Anomaly Detection},
  author={Roth et al. (2021)},
  year={2021},
  note={arXiv:2106.08265}
}

• arXiv: 2106.08265