Nplanqa Eval

nplanqa-eval

NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models — Park et al. (2025) (arXiv:2503.12772, 2025)

What this evaluates

This benchmark evaluates multi-modal large language models on their ability to understand complex driving scenes using multi-view and multi-frame inputs. It probes three core capabilities: road environment perception, spatial relations recognition, and ego-centric reasoning through visual question answering.

Datasets

• NuPlanQA-Eval — total ?; splits: test (-1); repo https://github.com/sungyeonparkk/NuPlanQA

Metrics

• accuracy (primary) — range: percent
  • Calculated as the number of correct responses divided by the total number of questions, multiplied by 100 to yield a percentage. The benchmark uses multiple-choice questions with four answer options, establishing a 25% random guessing baseline.

Input / output format

Input: Structured multi-view images (arranged into a single composite image with camera view markers) or single front-view images, optionally combined with historical frames (up to 3 past frames for ~1.5s of video). Accompanied by a multiple-choice question with four options.

Output: A single selected answer choice from the four provided options.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models frequently fail to prioritize the correct traffic light when multiple signals are present in the scene.
• Spatial relations recognition is significantly harder than other tasks because it relies purely on visual cues and temporal reasoning across frames, lacking the aid of control signals or static pretraining biases.
• Performance drops substantially when models are restricted to single-frame inputs, as temporal context is critical for ego-centric reasoning and spatial tracking.

Evidence (verbatim from paper)

Since our benchmark consists of multiple-choice questions with four answer options, we adopt accuracy as the evaluation metric, where random guessing yields a baseline accuracy of 25%.

Citation

@misc{park2025nplanqa,
  title={NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2503.12772}
}

• arXiv: 2503.12772