Padetbench Eval

padetbench-eval

PADetBench: Towards Benchmarking Physical Attacks against Object Detection — Lian et al. (2024) (arXiv:2408.09181, 2024)

What this evaluates

Evaluates the robustness of object detectors against various physical-world adversarial attacks in a controlled simulation environment. It measures how effectively different attack methods degrade detection performance across multiple object categories and detector architectures under strictly aligned physical dynamics.

Datasets

• PADetBench — total ?; splits: test (-1); repo https://github.com/JiaweiLian/PADetBench

Metrics

• ASR (Attack Success Rate) (primary) — range: percent
  • 1 - mAR@50 (mean Average Recall at IoU threshold 0.5). Measures the proportion of attacks that successfully degrade detector recall, reported as a percentage.

Input / output format

Input: Simulated images containing target objects (vehicles, persons, traffic signs) under controlled physical dynamics (weather, altitude, azimuth, radius, spawn points) with applied physical perturbations or attacks.

Output: Detection bounding boxes and class labels per image, processed by 48 different object detectors.

Scoring recipe

def compute_asr(predictions, gold, iou_thresh=0.5):
    # Calculate mean Average Recall at IoU threshold 0.5
    mAR = calculate_mean_average_recall(predictions, gold, iou=iou_thresh)
    # Attack Success Rate is the complement of recall
    asr = 1.0 - mAR
    return asr * 100  # Return as percentage

Common pitfalls

• Failing to strictly align physical dynamics (weather, camera angle, distance) across evaluations, which previous works often underestimate and leads to unfair comparisons.
• Assuming 2D adversarial patches transfer directly to 3D physical environments, ignoring the significant performance gap highlighted in the ablation studies.
• Equating state-of-the-art detection accuracy with adversarial robustness, as the benchmark shows SOTA detectors are not necessarily more robust to physical attacks.

Evidence (verbatim from paper)

ASR is measured by mAR50. We generate overall datasets with three objects, ten weather conditions, two altitude angles, eight azimuth angles, five radius values, three spawn points, and 23 physical perturbations, i.e., 7200 samples ($3 imes 10 imes 2 imes 8 imes 5 imes 3=7200$) for each attack method, in which the physical dynamics are strictly aligned and controlled for impartial comparison as shown in Fig. [3].

Citation

@misc{lian2024padetbench,
  title={PADetBench: Towards Benchmarking Physical Attacks against Object Detection},
  author={Lian et al. (2024)},
  year={2024},
  note={arXiv:2408.09181}
}

• arXiv: 2408.09181