Ad4ad Eval

ad4ad-eval

AD4AD: Benchmarking Visual Anomaly Detection Models for Safer Autonomous Driving — Genilotti et al. (2026) (arXiv:2604.15291, 2026)

What this evaluates

Evaluates visual anomaly detection models for autonomous driving by measuring their ability to detect and precisely localize defects or hazards in road scenes. It probes the trade-off between detection accuracy, pixel-level localization precision, and computational efficiency for onboard deployment.

Datasets

• AD4AD (AnoVox) — total ?; splits: test (-1)

Metrics

• I-ROC — range: [0, 1]
  • Area under the receiver operating characteristic curve computed at the image level using anomaly scores and binary labels.
• I-F1 — range: [0, 1]
  • Harmonic mean of image-level precision and recall, calculated from image-level anomaly scores against a fixed threshold.
• P-AP (primary) — range: [0, 1]
  • Area under the precision-recall curve computed at the pixel level, comparing predicted anomaly heatmaps against ground-truth pixel masks.
• P-F1 — range: [0, 1]
  • Harmonic mean of pixel-level precision and recall, calculated from thresholded anomaly maps against ground-truth pixel masks.

Input / output format

Input: RGB images of road scenes (synthetic and real-world driving conditions).

Output: Image-level anomaly scores for detection, and pixel-level anomaly heatmaps/masks for localization.

Scoring recipe

def compute_metrics(predictions, gold):
    # Image-level
    img_scores = [p['img_score'] for p in predictions]
    img_labels = [g['img_label'] for g in gold]
    i_roc = roc_auc_score(img_labels, img_scores)
    i_f1 = f1_score(img_labels, (np.array(img_scores) > 0.5).astype(int))
    
    # Pixel-level
    p_aps, p_f1s = [], []
    for pred, gt in zip(predictions, gold):
        p_map = pred['pixel_map'].flatten()
        g_mask = gt['pixel_mask'].flatten()
        p_aps.append(average_precision_score(g_mask, p_map))
        p_f1s.append(f1_score(g_mask, (p_map > 0.5).astype(int)))
    return {'I-ROC': i_roc, 'I-F1': i_f1, 'P-AP': np.mean(p_aps), 'P-F1': np.mean(p_f1s)}

Common pitfalls

• Relying solely on I-ROC, which remains high even when models fail to detect anomalies under class imbalance; I-F1 is a more reliable indicator of practical performance.
• Ignoring spatial context requirements: models using memory banks without positional awareness (e.g., PatchCore) perform poorly on structured road scenes where anomalies are localized objects.
• Overlooking scale and geometric sensitivity: patch-based methods struggle with tiny/distant objects and perspective distortions from road curves, leading to fragmented or missed anomaly maps.

Evidence (verbatim from paper)

At the image level, most models achieve a very high I-ROC; instead, by examining the more challenging I-F1 score, which better reflects practical detection performance under class imbalance, a different picture emerges. Some models such as FastFlow (0.97), PaDiM (0.91), PatchCore (0.95), and Dinomaly (0.94) stand out as the strongest performers with DeiT-Small. In contrast, STFPM (0.73) and SSNet (0.38) lag considerably behind, suggesting that despite acceptable ROC scores, these methods struggle to maintain a reliable performance at the image level. While many models show a generally strong ability to discriminate between normal and anomalous images, when considering pixel-level localization, measured by P-AP, the gap between methods becomes even more pronounced.

Citation

@misc{genilotti2026ad4ad,
  title={AD4AD: Benchmarking Visual Anomaly Detection Models for Safer Autonomous Driving},
  author={Genilotti et al. (2026)},
  year={2026},
  note={arXiv:2604.15291}
}

• arXiv: 2604.15291