Mvtec Eval

mvtec-eval

CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows — Gudovskiy et al. (2021) (arXiv:2107.12571, 2021)

What this evaluates

Unsupervised anomaly detection and pixel-level localization on industrial defect data. It probes the model's ability to distinguish normal from defective samples and precisely segment defect regions without using labeled anomalies during training.

Datasets

• MVTec — total 5354; splits: train (3629), test (1725)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Computed for both image-level detection scores and flattened pixel-level anomaly score maps.
• AUPRO — range: [0, 1]
  • Area under the per-region-overlap curve. Computes the average true positive rate across multiple overlap thresholds between predicted and ground truth masks, ensuring equal importance for large and small anomalies.

Input / output format

Input: RGB images resized to class-specific resolutions (128x128, 256x256, or 512x512) without cropping. Training images undergo ±5° rotation augmentation.

Output: Per-image anomaly score for detection. Per-pixel anomaly score map for localization, optionally thresholded to produce binary segmentation masks.

Scoring recipe

img_scores = model.predict_image_scores(images)
auroc_det = compute_auroc(img_scores, gt_labels)

score_maps = model.predict_anomaly_maps(images)
auroc_loc = compute_auroc(score_maps.flatten(), gt_pixel_labels)
aupro_loc = compute_aupro(score_maps, gt_masks, thresholds=[0.1, 0.5])

Common pitfalls

• AUROC is skewed towards large-area anomalies; AUPRO must be reported alongside to fairly evaluate small defect localization.
• Optimal input resolution varies by class (e.g., 256x256 for macro objects, 512x512 for most, 128x128 for transistor); using a fixed resolution degrades performance.
• Training is strictly unsupervised; only anomaly-free images are used. Using defect labels during training violates the protocol.

Evidence (verbatim from paper)

We use widely-used threshold-agnostic evaluation metrics for localization: area under the receiver operating characteristic curve (AUROC) and area under the per-region-overlap curve (AUPRO). AUROC is skewed towards large-area anomalies, while AUPRO metric ensures that both large and small anomalies are equally important in localization. Image-level AD detection is reported by the AUROC only.

Citation

@misc{gudovskiy2021cflowad,
  title={CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows},
  author={Gudovskiy et al. (2021)},
  year={2021},
  note={arXiv:2107.12571}
}

• arXiv: 2107.12571