Real Iad Eval

real-iad-eval

Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection — Wang et al. (2024) (arXiv:2403.12580, 2024)

What this evaluates

Evaluates industrial anomaly detection models under standard unsupervised and fully unsupervised (noisy training) settings. It probes image-level, pixel-level, and multi-view sample-level defect detection capabilities.

Datasets

• Real-IAD — total 150000; splits: train (-1), test (200)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, computed separately for image-level and pixel-level anomaly scores.
• AUPRO — range: [0, 1]
  • Area Under the Per-Region Overlap curve, measuring pixel-level segmentation accuracy against ground truth masks.

Input / output format

Input: High-resolution industrial product images, optionally provided as multiple views (up to 5 angles) per sample.

Output: Per-image and per-pixel anomaly scores, or binary segmentation masks indicating defective regions.

Scoring recipe

def compute_auroc(y_true, y_scores):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

def compute_aupro(pred_masks, gt_masks):
    pros = []
    for pred, gt in zip(pred_masks, gt_masks):
        overlap = compute_intersection_over_union(pred, gt)
        pros.append(overlap)
    return compute_auc(pros)

# Multi-view aggregation: average or max scores across views per sample
sample_scores = aggregate_across_views(image_scores, view_scores)

Common pitfalls

• FUIAD training sets must be constructed with a specific noise ratio (α) rather than using all available anomalies.
• Test set is strictly fixed to 100 normal and 100 anomalous samples (500 images total across views); do not use the full dataset for testing.
• Multi-view performance requires aggregating scores across views to match industrial production line evaluation, not just reporting per-view metrics.

Evidence (verbatim from paper)

The AUROC is the most widely used metric for image-level and pixel-level anomaly detection. In addition, a normalized Per-Region Overlap (PRO) between segmentation and ground truth is calculated and the Area Under PRO curve (AUPRO) is also adopted as a pixel-level metric. To create an FUIAD setting, we first need to fix the testing set including normal and anomaly samples. In our experiment, the number of normal and anomaly samples is uniformly set to 100 samples (500 images).

Citation

@misc{wang2024realiad,
  title={Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2403.12580}
}

• arXiv: 2403.12580