Nuplan R Eval

nuplan-r-eval

nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous Driving via Reactive Multi-Agent Simulation — Peng et al. (2025) (arXiv:2511.10403, 2025)

What this evaluates

Evaluates autonomous driving planners in a closed-loop setting with realistic, reactive multi-agent traffic. It probes a planner's ability to handle complex, interactive driving scenarios by measuring overall success, robustness against catastrophic failures, and consistency across safety and comfort dimensions.

Datasets

• nuPlan-R — total ?; splits: Test14-Random (-1), Test14-Hard (-1)

Metrics

• CLS (primary) — range: percent
  • Standard Closed-Loop Success metric from the nuPlan benchmark, evaluating overall planner performance across safety, comfort, and rule compliance during closed-loop simulation.
• SR — range: percent
  • Success Rate, a binary or percentage metric indicating whether a planner reliably avoids catastrophic failures (e.g., collisions, severe violations) during closed-loop simulation.
• PR — range: percent
  • All-Core Pass Rate, measuring the fraction of core evaluation dimensions (e.g., safety, comfort, traffic rules) that a planner passes consistently across the scenario.

Input / output format

Input: Scenario state including ego vehicle dynamics, surrounding traffic agents, and high-definition map features (e.g., vectorized polylines or rasterized BEV images).

Output: A planned trajectory for the ego vehicle over the simulation horizon, typically propagated via a kinematic bicycle model with LQR tracking.

Scoring recipe

def evaluate_nuplan_r(planner, scenario, reactive_agents):
    trajectory = run_closed_loop(planner, scenario, reactive_agents)
    cls = compute_cls(trajectory)
    sr = 1.0 if not has_catastrophic_failure(trajectory) else 0.0
    pr = compute_core_pass_rate(trajectory)
    return {'CLS': cls, 'SR': sr, 'PR': pr}

Common pitfalls

• Using the original nuPlan's rule-based IDM agents instead of nuPlan-R's reactive diffusion-based agents, which artificially inflates rule-based planner scores.
• Relying exclusively on CLS, which can mask planner fragility or unbalanced behavior; SR and PR are required to detect catastrophic failure sensitivity and cross-dimensional consistency.
• Evaluating in open-loop mode; nuPlan-R requires frame-by-frame closed-loop simulation with reactive agent updates to capture multi-directional interactions.

Evidence (verbatim from paper)

In the original nuPlan benchmark, surrounding vehicles are controlled by a rule-based reactive agent based on the IDM, while in our nuPlan-R benchmark, they are driven by the proposed learning-based reactive agent, which enables more realistic interactions. In addition to the standard CLS metric, we also report two additional metrics, namely SR and PR metrics, as described in Section[III-C]. In all closed-loop simulations, we further employ a Linear Quadratic Regulator (LQR) controller for trajectory tracking, and the resulting control outputs are applied to a kinematic bicycle model to propagate the vehicle dynamics.

Citation

@misc{peng2025nuplanr,
  title={nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous Driving via Reactive Multi-Agent Simulation},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2511.10403}
}

• arXiv: 2511.10403