Drivingvqa Eval

drivingvqa-eval

Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios — Corbière et al. (2025) (arXiv:2501.04671, 2025)

What this evaluates

Evaluates a vision-language model's ability to perform multi-label multiple-choice question answering on real-world driving scenarios, requiring precise visual grounding and spatial reasoning to select all correct answers from a set of options.

Datasets

• DrivingVQA — total ?; splits: train (-1), test (-1)

Metrics

• exam score (primary) — range: percent
  • Equivalent to exact-match accuracy for multi-label classification. A question scores 1 only if all correct answer options are selected, and 0 otherwise. The final score is the percentage of correctly answered questions.
• F1-Score — range: percent
  • Multi-label classification F1-score computed over all answer options across the test set.

Input / output format

Input: Image, question text, and list of possible answers. Optionally augmented with entity labels, bounding box coordinates, and/or visual crops of relevant entities.

Output: Predicted answer(s), optionally preceded by a chain-of-thought explanation. For RIV-CoT, the output interleaves entity bounding boxes, visual patches, and reasoning steps before the final answer.

Scoring recipe

def compute_exam_score(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if set(p) == set(g))
    return (correct / len(predictions)) * 100

Common pitfalls

• The exam score is an exact-match metric for multi-label answers, not standard single-label accuracy. Missing or adding any option results in a 0 for that question.
• Results are averaged over 5 random seeds and reported with standard deviation; single-run evaluations will not match paper tables.
• Visual inputs are strictly required; evaluating without images causes a ~26 point drop, so text-only baselines are invalid for this benchmark.

Evidence (verbatim from paper)

Evaluation metrics. The performance on the DrivingVQA test split is measured with the exam score, analogous to the real driving theory score used to evaluate candidates: all correct answers must be selected to score a question correctly. Note that this score is equivalent to the exact match ratio. Since the task is analogous to multi-label classification, we also report the F1-Score.

Citation

@misc{corbiere2025rivcot,
  title={Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios},
  author={Corbière et al. (2025)},
  year={2025},
  note={arXiv:2501.04671}
}

• arXiv: 2501.04671