# Drivingvqa Eval

> Evaluates a vision-language model's ability to perform multi-label multiple-choice question answering on real-world driving scenarios, requiring precise visual grounding and spatial reasoning to select all correct answers from a set of options. Use when the user wants to benchmark on DrivingVQA, or asks about evaluating this task. Reports exam score.

- Skill: `qhjqhj00/drivingvqa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/drivingvqa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/drivingvqa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/drivingvqa-eval

---


# drivingvqa-eval

> Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios — Corbière et al. (2025) (arXiv:2501.04671, 2025)

## What this evaluates

Evaluates a vision-language model's ability to perform multi-label multiple-choice question answering on real-world driving scenarios, requiring precise visual grounding and spatial reasoning to select all correct answers from a set of options.

## Datasets

- **DrivingVQA** — total ?; splits: train (-1), test (-1)

## Metrics

- `exam score` **(primary)** — range: percent
  - Equivalent to exact-match accuracy for multi-label classification. A question scores 1 only if all correct answer options are selected, and 0 otherwise. The final score is the percentage of correctly answered questions.
- `F1-Score` — range: percent
  - Multi-label classification F1-score computed over all answer options across the test set.

## Input / output format

**Input**: Image, question text, and list of possible answers. Optionally augmented with entity labels, bounding box coordinates, and/or visual crops of relevant entities.

**Output**: Predicted answer(s), optionally preceded by a chain-of-thought explanation. For RIV-CoT, the output interleaves entity bounding boxes, visual patches, and reasoning steps before the final answer.

## Scoring recipe

```python
def compute_exam_score(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if set(p) == set(g))
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- The exam score is an exact-match metric for multi-label answers, not standard single-label accuracy. Missing or adding any option results in a 0 for that question.
- Results are averaged over 5 random seeds and reported with standard deviation; single-run evaluations will not match paper tables.
- Visual inputs are strictly required; evaluating without images causes a ~26 point drop, so text-only baselines are invalid for this benchmark.

## Evidence (verbatim from paper)

> Evaluation metrics. The performance on the DrivingVQA test split is measured with the exam score, analogous to the real driving theory score used to evaluate candidates: all correct answers must be selected to score a question correctly. Note that this score is equivalent to the exact match ratio. Since the task is analogous to multi-label classification, we also report the F1-Score.

## Citation

```bibtex
@misc{corbiere2025rivcot,
  title={Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios},
  author={Corbière et al. (2025)},
  year={2025},
  note={arXiv:2501.04671}
}
```

- arXiv: 2501.04671