# Padetbench Eval

> Evaluates the robustness of object detectors against various physical-world adversarial attacks in a controlled simulation environment. It measures how effectively different attack methods degrade detection performance across multiple object categories and detector architectures under strictly aligned physical dynamics. Use when the user wants to benchmark on PADetBench, or asks about evaluating this task. Reports ASR (Attack Success Rate).

- Skill: `qhjqhj00/padetbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/padetbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/padetbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/padetbench-eval

---


# padetbench-eval

> PADetBench: Towards Benchmarking Physical Attacks against Object Detection — Lian et al. (2024) (arXiv:2408.09181, 2024)

## What this evaluates

Evaluates the robustness of object detectors against various physical-world adversarial attacks in a controlled simulation environment. It measures how effectively different attack methods degrade detection performance across multiple object categories and detector architectures under strictly aligned physical dynamics.

## Datasets

- **PADetBench** — total ?; splits: test (-1); repo https://github.com/JiaweiLian/PADetBench

## Metrics

- `ASR (Attack Success Rate)` **(primary)** — range: percent
  - 1 - mAR@50 (mean Average Recall at IoU threshold 0.5). Measures the proportion of attacks that successfully degrade detector recall, reported as a percentage.

## Input / output format

**Input**: Simulated images containing target objects (vehicles, persons, traffic signs) under controlled physical dynamics (weather, altitude, azimuth, radius, spawn points) with applied physical perturbations or attacks.

**Output**: Detection bounding boxes and class labels per image, processed by 48 different object detectors.

## Scoring recipe

```python
def compute_asr(predictions, gold, iou_thresh=0.5):
    # Calculate mean Average Recall at IoU threshold 0.5
    mAR = calculate_mean_average_recall(predictions, gold, iou=iou_thresh)
    # Attack Success Rate is the complement of recall
    asr = 1.0 - mAR
    return asr * 100  # Return as percentage
```

## Common pitfalls

- Failing to strictly align physical dynamics (weather, camera angle, distance) across evaluations, which previous works often underestimate and leads to unfair comparisons.
- Assuming 2D adversarial patches transfer directly to 3D physical environments, ignoring the significant performance gap highlighted in the ablation studies.
- Equating state-of-the-art detection accuracy with adversarial robustness, as the benchmark shows SOTA detectors are not necessarily more robust to physical attacks.

## Evidence (verbatim from paper)

> ASR is measured by mAR50. We generate overall datasets with three objects, ten weather conditions, two altitude angles, eight azimuth angles, five radius values, three spawn points, and 23 physical perturbations, i.e., 7200 samples ($3	imes 10	imes 2	imes 8	imes 5	imes 3=7200$) for each attack method, in which the physical dynamics are strictly aligned and controlled for impartial comparison as shown in Fig. [3].

## Citation

```bibtex
@misc{lian2024padetbench,
  title={PADetBench: Towards Benchmarking Physical Attacks against Object Detection},
  author={Lian et al. (2024)},
  year={2024},
  note={arXiv:2408.09181}
}
```

- arXiv: 2408.09181