# Dsbench Eval

> Evaluates Vision-Language Models' ability to perceive and reason about safety-critical scenarios in autonomous driving, covering both external environmental hazards (e.g., traffic rules, obstacles, weather) and in-cabin driver states (e.g., fatigue, distraction, emotion). It probes fine-grained hazard recognition, regulatory compliance, and multi-step safety reasoning under diverse, high-risk conditions. Use when the user wants to benchmark on DSBench, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/dsbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/dsbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/dsbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/dsbench-eval

---


# dsbench-eval

> Is Your VLM for Autonomous Driving Safety-Ready? A Comprehensive Benchmark for Evaluating External and In-Cabin Risks — Meng et al. (2025) (arXiv:2511.14592, 2025)

## What this evaluates

Evaluates Vision-Language Models' ability to perceive and reason about safety-critical scenarios in autonomous driving, covering both external environmental hazards (e.g., traffic rules, obstacles, weather) and in-cabin driver states (e.g., fatigue, distraction, emotion). It probes fine-grained hazard recognition, regulatory compliance, and multi-step safety reasoning under diverse, high-risk conditions.

## Datasets

- **DSBench** — total 3000; splits: test (3000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered QA pairs. Predictions are compared against ground-truth answers using exact-match or semantic similarity scoring, averaged across all 28 safety subcategories.

## Input / output format

**Input**: Image(s) of driving scenes (external road views and/or in-cabin cockpit views) paired with a natural language question targeting a specific safety dimension or subcategory.

**Output**: Natural language answer (short phrase or sentence) indicating the correct safety action, hazard identification, or driver state classification.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_text(pred) == normalize_text(gold):
        correct += 1
return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models often fail on hypothetical or multi-step reasoning questions that require chaining safety logic, rather than simple object detection.
- In-cabin and external scene confusion: Models trained on siloed datasets may misattribute driver states to external factors or vice versa.
- Pre-annotation bias: The dataset uses GPT-4o for initial categorization, which may propagate specific reasoning patterns or blind spots to evaluated models.

## Evidence (verbatim from paper)

> For evaluation, we curate a subset of 3,000 representative high-risk scenes to form DSBench. In total, we construct 98K QAs covering 28 safety dimensions, combining scale with diversity.

## Citation

```bibtex
@misc{meng2025dsbench,
  title={Is Your VLM for Autonomous Driving Safety-Ready? A Comprehensive Benchmark for Evaluating External and In-Cabin Risks},
  author={Meng et al. (2025)},
  year={2025},
  note={arXiv:2511.14592}
}
```

- arXiv: 2511.14592