Dsbench Eval

dsbench-eval

Is Your VLM for Autonomous Driving Safety-Ready? A Comprehensive Benchmark for Evaluating External and In-Cabin Risks — Meng et al. (2025) (arXiv:2511.14592, 2025)

What this evaluates

Evaluates Vision-Language Models' ability to perceive and reason about safety-critical scenarios in autonomous driving, covering both external environmental hazards (e.g., traffic rules, obstacles, weather) and in-cabin driver states (e.g., fatigue, distraction, emotion). It probes fine-grained hazard recognition, regulatory compliance, and multi-step safety reasoning under diverse, high-risk conditions.

Datasets

• DSBench — total 3000; splits: test (3000)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered QA pairs. Predictions are compared against ground-truth answers using exact-match or semantic similarity scoring, averaged across all 28 safety subcategories.

Input / output format

Input: Image(s) of driving scenes (external road views and/or in-cabin cockpit views) paired with a natural language question targeting a specific safety dimension or subcategory.

Output: Natural language answer (short phrase or sentence) indicating the correct safety action, hazard identification, or driver state classification.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_text(pred) == normalize_text(gold):
        correct += 1
return (correct / len(gold_answers)) * 100

Common pitfalls

• Models often fail on hypothetical or multi-step reasoning questions that require chaining safety logic, rather than simple object detection.
• In-cabin and external scene confusion: Models trained on siloed datasets may misattribute driver states to external factors or vice versa.
• Pre-annotation bias: The dataset uses GPT-4o for initial categorization, which may propagate specific reasoning patterns or blind spots to evaluated models.

Evidence (verbatim from paper)

For evaluation, we curate a subset of 3,000 representative high-risk scenes to form DSBench. In total, we construct 98K QAs covering 28 safety dimensions, combining scale with diversity.

Citation

@misc{meng2025dsbench,
  title={Is Your VLM for Autonomous Driving Safety-Ready? A Comprehensive Benchmark for Evaluating External and In-Cabin Risks},
  author={Meng et al. (2025)},
  year={2025},
  note={arXiv:2511.14592}
}

• arXiv: 2511.14592