Dvbench Eval

dvbench-eval

Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding — Zeng et al. (2025) (arXiv:2504.14526, 2025)

What this evaluates

Evaluates Vision Large Language Models' ability to understand safety-critical driving videos across a hierarchical taxonomy of 25 abilities, including perception, temporal-spatial reasoning, and risk assessment.

Datasets

• DVBench — total ?; splits: test (-1); repo https://github.com/tong-zeng/DVBench.git

Metrics

• Top-1 accuracy (primary) — range: percent
  • Calculated under the GroupEval protocol: each multiple-choice question is presented N times (N=4). The correct answer's position is rotated systematically while other options are shuffled. A question is counted as correct only if the model selects the right answer in all N trials. Final score is the percentage of questions passed.

Input / output format

Input: A driving video clip followed by a multiple-choice question (typically 4 options) probing safety-critical understanding.

Output: A single selected option (e.g., A, B, C, or D) per trial.

Scoring recipe

def compute_group_eval_accuracy(predictions, golds, n_trials=4):
    passed = 0
    for q_preds, q_gold in zip(predictions, golds):
        if all(p == q_gold for p in q_preds):
            passed += 1
    return (passed / len(golds)) * 100

Common pitfalls

• Position bias: models often favor specific answer positions (e.g., first or last), inflating single-trial accuracy.
• Single-trial evaluation (IndividualEval) significantly overestimates performance compared to the more rigorous GroupEval protocol.
• Random guessing baseline of ~25% for 4-choice questions masks true capability gaps.

Evidence (verbatim from paper)

DVBench employs a multiple-choice question format, which introduces inherent evaluation challenges. Random guessing could yield approximately 25% top-1 accuracy for 4-choice questions, potentially diminishing the observable performance differences among VLLMs. Furthermore, systematic biases in VLLMs complicate accurate performance assessment. For instance, many VLLMs exhibit position bias, disproportionately favoring specific answer positions, such as always selecting the first or last option*(Liu et al., [2024b]; Chen et al., [2024])*. This tendency is further corroborated by Figure[5], highlighting the complexities of evaluating VLLMs fairly and reliably. To address these challenges, we propose Group Evaluation (GroupEval), a more robust strategy. Each question is presented to VLLMs $N$ times (where $N$ equals the number of answer choices), with correct answer’s position rotates systematically while other options are randomly shuffled (as illustrated in Figure[4]). A VLLM passes a question only if it identifies the correct answer in all $N$ trials.

Citation

@misc{zeng2025dvbench,
  title={Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding},
  author={Zeng et al. (2025)},
  year={2025},
  note={arXiv:2504.14526}
}

• arXiv: 2504.14526