Im Iad Eval

im-iad-eval

IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing — Xie et al. (2023) (arXiv:2301.13359, 2023)

What this evaluates

Evaluates industrial image anomaly detection algorithms across seven manufacturing datasets under unsupervised, few-shot, continual, and fully supervised settings. It probes both image-level classification and pixel-level localization capabilities, while also measuring computational efficiency like inference speed and GPU memory.

Datasets

• MVTec AD — total ?; splits: test (-1)
• MVTec LOCO-AD — total ?; splits: test (-1)
• MPDD — total ?; splits: test (-1)
• BTAD — total ?; splits: test (-1)
• MTD — total ?; splits: test (-1)
• VisA — total ?; splits: test (-1)
• DAGM — total ?; splits: test (-1)

Metrics

• Image AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve for image-level binary anomaly classification. Computed by varying the classification threshold and plotting true positive rate against false positive rate.
• Pixel AP — range: [0, 1]
  • Average Precision for pixel-level anomaly localization. Computed from the precision-recall curve over all pixels using the anomaly heatmap scores.
• Pixel PRO — range: [0, 1]
  • Product-Region Overlap metric for pixel-level localization. Measures the mean overlap ratio between predicted anomaly regions and ground truth masks across multiple thresholds.

Input / output format

Input: RGB images of industrial products, accompanied by binary masks for pixel-level anomalies and binary labels for image-level classification.

Output: Per-image anomaly score and per-pixel anomaly heatmap/mask.

Scoring recipe

def compute_auc(y_true, y_pred):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

def compute_ap(y_true, y_pred):
    precision, recall, _ = precision_recall_curve(y_true, y_pred)
    return auc(recall, precision)

def compute_pro(y_true_mask, y_pred_map, thresholds):
    overlaps = []
    for thr in thresholds:
        pred_mask = (y_pred_map > thr).astype(int)
        intersection = np.logical_and(pred_mask, y_true_mask).sum()
        union = np.logical_or(pred_mask, y_true_mask).sum()
        overlaps.append(intersection / union if union > 0 else 0)
    return np.mean(overlaps)

Common pitfalls

• Image-level and pixel-level metrics often diverge; a model can excel at one but fail at the other, so reporting only one is misleading.
• Logical anomalies (displacement/missing parts) are structurally different from dents/scratches and require global feature extraction that many structural detectors lack.
• Efficiency metrics (inference time, GPU memory) are critical for real-world deployment but are frequently secondary to accuracy in academic benchmarks.

Evidence (verbatim from paper)

Researchers commonly use image-level and pixel-level metrics to evaluate the classification performance of IAD algorithms. In practice, the image-level metric is used to judge whether the whole product is abnormal, while the pixel-level metric indicates anomaly localization performance... According to Table VI, specific IAD methods, like PatchCore, perform well on image AUROC but poorly on pixel AP, or vice versa.

Citation

@misc{xie2023imiad,
  title={IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing},
  author={Xie et al. (2023)},
  year={2023},
  note={arXiv:2301.13359}
}

• arXiv: 2301.13359