Rad Robustness Eval

rad-robustness-eval

RAD: A Comprehensive Dataset for Benchmarking the Robustness of Image Anomaly Detection — Cheng et al. (2024) (arXiv:2406.07176, 2024)

What this evaluates

Evaluates the robustness of image anomaly detection models against real-world imaging distortions, including free viewpoints, uneven illumination, and motion blur. It measures how well unsupervised and zero-shot methods localize and classify anomalies on industrial work platforms with foreign objects.

Datasets

• RAD — total ?; splits: test (-1); repo https://github.com/hustCYQ/RAD-dataset

Metrics

• AUROC (primary) — range: percent
  • Area Under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
• Max-F1 — range: percent
  • Maximal F1 score. The highest F1 score achieved across all possible classification thresholds on the predicted anomaly scores.
• AP — range: percent
  • Average Precision. The area under the Precision-Recall curve, summarizing model performance across all thresholds.
• AUPRO — range: percent
  • Area Under Per Region Overlap curve. Evaluates pixel-level localization by computing the overlap ratio between predicted and ground truth anomaly masks for each image, then averaging across the dataset.

Input / output format

Input: Resized 224×224 RGB images of industrial work platforms containing foreign objects (bolts, cable ties, sponges, tapes) under varying conditions (normal, uneven illumination, blurry, free views).

Output: Image-level anomaly score/prediction and pixel-wise anomaly score map or binary mask.

Scoring recipe

# Image-level
auroc = roc_auc_score(y_true_img, y_score_img)
ap = average_precision_score(y_true_img, y_score_img)
max_f1 = max(f1_score(y_true_img, (y_score_img > t)) for t in np.linspace(0, 1, 100))
# Pixel-level
aupro = compute_aupro(y_true_masks, y_score_maps) # AUPRO averages per-image overlap ratios

Common pitfalls

• AUPRO is exclusively for pixel-level evaluation and should not be reported for image-level results.
• All input images are resized to 224×224, which may degrade pixel-level localization precision compared to original high-resolution captures.
• Zero-shot methods (WinCLIP, APRIL-GAN, SAA) are evaluated without any training on the RAD dataset, relying entirely on pre-trained foundation models.

Evidence (verbatim from paper)

Following the mainstream setting as in MVTec-AD, we employ several key metrics to evaluate the performance of anomaly detection methods, namely the Area Under the Receiver Operating Characteristic curve (AUROC), Maximal F1 score (Max-F1), Average Precision (AP), and Area Under Per Region Overlap (AUPRO). The first three indicators (AUROC, Max-F1, AP) are utilized for both image-level and pixel-level anomaly detection evaluation, and AUPRO is exclusively employed for pixel-level anomaly detection evaluation.

Citation

@misc{cheng2024rad,
  title={RAD: A Comprehensive Dataset for Benchmarking the Robustness of Image Anomaly Detection},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2406.07176}
}

• arXiv: 2406.07176