Nuplan Planning Eval

nuplan-planning-eval

Driving is a Game: Combining Planning and Prediction with Bayesian Iterative Best Response — Distelzweig et al. (2025) (arXiv:2512.03936, 2025)

What this evaluates

Evaluates a driving planner's ability to navigate interactive scenarios in a closed-loop simulator. It measures success rates under both non-reactive (log-replay) and reactive (IDM/SMART) traffic conditions across routine validation splits and complex, human-curated scenarios.

Datasets

• Val14 — total 1118; splits: val (1118)
• Test14 — total ?; splits: random (-1), hard (280)
• interPlan — total 80; splits: full (80), lane-change (30)

Metrics

• CLS-NR (primary) — range: percent
  • Closed-loop success score computed by running the planner in the nuPlan simulator over the full scenario horizon with non-reactive surrounding agents that follow pre-recorded trajectories.
• CLS-R (primary) — range: percent
  • Closed-loop success score computed under reactive surrounding agents using either the Intelligent Driver Model (IDM) or the SMART model to respond to ego behavior.

Input / output format

Input: Historical agent trajectories $X_{1:N_a}^{-t_h:0}$ and map context $C_{\mathrm{map}}$.

Output: Ego trajectory $Y_{\mathrm{ego}}^t$ sampled at 10 Hz over a 4-second future horizon (40 positions).

Scoring recipe

def compute_nuplan_score(planner, scenario, agent_mode='NR'):
    sim = nuPlanSimulator(scenario, agent_mode=agent_mode)
    planner.run(sim)
    success_rate = sim.evaluate_closed_loop()
    return success_rate * 100

Common pitfalls

• Confusing non-reactive (NR) and reactive (R) settings: NR replays logged trajectories, while R uses IDM or SMART models to react to the ego vehicle.
• Assuming Test14-hard is randomly sampled: it is curated by running PDM-Closed and selecting the 20 most challenging cases per category.
• Treating the metric as open-loop trajectory matching: evaluation is strictly closed-loop, meaning planning errors compound over the 4s horizon.

Evidence (verbatim from paper)

We report closed-loop non-reactive (CLS-NR) and closed-loop reactive (CLS-R) scores for Val14, Test14-hard, and Test14-random splits, as well as reactive scores on the interPlan and interPlanLC scenarios. Higher values indicate better performance.

Citation

@misc{distelzweig2025drivingisagame,
  title={Driving is a Game: Combining Planning and Prediction with Bayesian Iterative Best Response},
  author={Distelzweig et al. (2025)},
  year={2025},
  note={arXiv:2512.03936}
}

• arXiv: 2512.03936