Bench2drive Vl Eval

bench2drive-vl-eval

Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models — Xiaosong Jia et al. (2026) (arXiv:2604.01259, 2026)

What this evaluates

Evaluates vision-language models in closed-loop autonomous driving by assessing their perception, prediction, planning, and behavioral reasoning capabilities within a CARLA simulator. It probes the model's ability to process raw sensor inputs, generate causally consistent natural language reasoning, and produce valid control actions across diverse and out-of-distribution driving scenarios.

Datasets

• Bench2Drive-VL — total ?; splits: test (-1); repo https://github.com/Thinklab-SJTU/Bench2Drive-VL

Metrics

• LLM-based rubric functions (primary) — range: [0, 1]
  • An LLM evaluator compares the VLM's predicted VQA answers against ground-truth responses generated by DriveCommenter. It applies rubric-based scoring to assess fine-grained perception, prediction, planning, and behavior reasoning quality.
• Bench2Drive metrics — range: percent
  • Quantitative closed-loop driving metrics applied to the VLM's control signals and trajectory within the CARLA simulator to evaluate planning quality, including success rate, collision avoidance, and lane adherence.

Input / output format

Input: Raw sensor inputs (images, BEV, text) transmitted by the CARLA simulator at each intervention interval, representing the current driving environment.

Output: Natural language reasoning, VQA answers, and an action selection (e.g., FOLLOW_LANE, CHANGE_LANE_LEFT, KEEP, STOP) that is converted into control signals for the simulator.

Scoring recipe

def evaluate_vlm(vlm_outputs, ground_truth, carla_state):
    vqa_scores = []
    for q_id, pred in vlm_outputs.vqa.items():
        gold = ground_truth[q_id]
        score = llm_rubric.evaluate(pred, gold)
        vqa_scores.append(score)
    
    control_signals = action_module.convert(vlm_outputs.action_selection)
    planning_score = bench2drive_metrics.evaluate(control_signals, carla_state)
    
    return {
        'vqa_accuracy': mean(vqa_scores),
        'planning_quality': planning_score,
        'overall': aggregate(vqa_scores, planning_score)
    }

Common pitfalls

• Confusing DriveCommenter (the expert model that generates ground-truth VQA labels) with the evaluated VLM policy; DriveCommenter is part of the evaluation pipeline, not the test subject.
• Assuming scoring is purely online; the protocol explicitly states that all expert answers and VLM predictions are stored and scoring is conducted offline to speed up the evaluation process.
• Overlooking that the benchmark covers out-of-distribution and recovery scenarios (e.g., off-road, blind spots, special role actors), which require specific handling beyond standard CARLA leaderboard metrics.

Evidence (verbatim from paper)

At each intervention interval, the CARLA simulator transmits privileged state information to DriveCommenter, an expert model that generates ground-truth answers for a complex set of vision-language questions based on current environment. Simultaneously, the evaluated VLM policy processes raw sensor inputs to produce its reasoning and control outputs. To speed up the evaluation process, all expert answers and VLM predictions are stored and the scoring is conducted offline. A dedicated VQA evaluator then compares these responses using LLM-based rubric functions, enabling fine-grained assessment of both perception and reasoning quality. Planning quality is subsequently evaluated using Bench2Drive metrics.

Citation

@misc{jia2026bench2drivevl,
  title={Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models},
  author={Xiaosong Jia et al. (2026)},
  year={2026},
  note={arXiv:2604.01259}
}

• arXiv: 2604.01259