Scenepilot Bench Eval

scenepilot-bench-eval

ScenePilot-Bench: A Large-Scale Dataset and Benchmark for Evaluation of Vision-Language Models in Autonomous Driving — Wang et al. (2026) (arXiv:2601.19582, 2026)

What this evaluates

Evaluates vision-language models on autonomous driving tasks, including scene understanding, spatial perception, and motion planning. It probes the models' ability to reason about driving scenarios, predict trajectories, and generalize across different geographic regions and traffic conventions.

Datasets

• ScenePilot-Bench — total ?; splits: train (200000), val (-1), test (100000); repo https://github.com/yjwangtj/ScenePilot-Bench

Metrics

• Overall Score (primary) — range: percent
  • Composite metric aggregating performance across Scene Understanding, Spatial Perception, Motion Planning, and GPT-Score. Exact weighting scheme is not specified in the excerpt.
• SPICE — range: [0, 1]
  • Scene Perception and Image Composition Evaluation metric used to measure semantic scene understanding quality.
• ADE — range: other
  • Average Displacement Error measuring the mean Euclidean distance between predicted and ground-truth trajectory points over the planning horizon.
• FDE@1 — range: other
  • Final Displacement Error at 1 second, measuring the Euclidean distance between the predicted and ground-truth endpoint after 1 second.
• GPT-Score — range: percent
  • LLM-based scoring metric that evaluates the quality, safety, and relevance of model-generated responses against gold references.

Input / output format

Input: Video clips paired with natural language questions (VQA format) probing scene understanding, spatial perception, and motion planning.

Output: Natural language text responses to the VQA questions.

Scoring recipe

def compute_metrics(predictions, gold):
    scene_score = compute_spice(predictions, gold) * 100
    risk_acc = compute_accuracy(predictions.risk, gold.risk) * 100
    spatial_acc = compute_accuracy(predictions.spatial, gold.spatial) * 100
    ade = compute_displacement_error(predictions.trajectory, gold.trajectory, metric='ade')
    fde1 = compute_displacement_error(predictions.trajectory, gold.trajectory, metric='fde', horizon=1)
    gpt_score = get_llm_score(predictions.text, gold.text) * 100
    # Overall Score aggregates sub-metrics; exact weights unspecified in text
    overall = (scene_score + risk_acc + spatial_acc + ade + fde1 + gpt_score) / 6
    return overall

Common pitfalls

• Data leakage between train and test sets if video clips overlap; the benchmark mitigates this by drawing samples independently from unique video splits.
• High sensitivity of motion planning metrics (e.g., DCS-Acc) to geographic and traffic-rule domain shifts, which can mask underlying perception capabilities and lead to over-optimistic generalization claims.
• GPT-Score variability due to LLM prompting differences and lack of standardized rubrics, making cross-study comparisons difficult without strict prompt control.

Evidence (verbatim from paper)

The results indicate that the model maintains a remarkably stable overall performance across unseen regions, with Overall Scores ranging from 67.17 to 70.32, reflecting strong geographic generalization.

Citation

@misc{wang2026scenepilotbench,
  title={ScenePilot-Bench: A Large-Scale Dataset and Benchmark for Evaluation of Vision-Language Models in Autonomous Driving},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2601.19582}
}

• arXiv: 2601.19582