Pad Eval

pad-eval

PAD: A Dataset and Benchmark for Pose-agnostic Anomaly Detection — Zhou et al. (2023) (arXiv:2310.07716, 2023)

What this evaluates

Evaluates the ability of anomaly detection models to identify and localize defects in 3D objects from unseen camera poses without requiring pose alignment. It probes pose-invariant representation learning and robustness to viewpoint changes in both pixel-level segmentation and image-level classification.

Datasets

• MAD — total 11000; splits: train (-1), test (-1); repo https://github.com/EricLee0224/PAD

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, calculated as the integral of the true positive rate (R_TP) with respect to the false positive rate (R_FP): AUROC = ∫(R_TP) dR_FP. Computed separately for pixel-level segmentation and image-level classification.

Input / output format

Input: RGB images of objects captured from multiple viewpoints (poses), containing either normal samples or samples with anomalies.

Output: Pixel-level anomaly segmentation masks and image-level anomaly classification scores/labels.

Scoring recipe

def compute_auroc(predictions, ground_truth):
    # predictions: flattened array of anomaly scores (pixel-level or image-level)
    # ground_truth: flattened array of binary labels (0 for normal, 1 for anomaly)
    fpr, tpr, _ = roc_curve(ground_truth, predictions)
    return auc(fpr, tpr)

Common pitfalls

• Reconstruction-based methods may fail to generate accurate normal references for unseen poses, leading to a disconnect between high AUROC scores and poor visual reconstruction quality.
• Image-level detection performance often degrades significantly in the pose-agnostic setting compared to pixel-level localization, as holistic normal features are harder to learn from sparse viewpoints.

Evidence (verbatim from paper)

Following previous work, we specifically choose the Area Under the Receiver Operating Caracteristic Curve (AUROC) as the primary metric for evaluating the performance of anomaly segmentation at the pixel-level and anomaly classification at the image-level. While there exist various evaluation metric for these tasks, AUROC stands out as the most widely used and suitable metric for conducting comprehensive benchmarking. The AUROC score can be calculated as follows: AUROC = ∫(R_TP) dR_FP

Citation

@misc{zhou2023pad,
  title={PAD: A Dataset and Benchmark for Pose-agnostic Anomaly Detection},
  author={Zhou et al. (2023)},
  year={2023},
  note={arXiv:2310.07716}
}

• arXiv: 2310.07716