Lead Closed Loop Eval

lead-closed-loop-eval

LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving — Long Nguyen et al. (arXiv:2512.20563, 2025)

What this evaluates

Evaluates the closed-loop driving performance and generalization of end-to-end autonomous driving policies in simulation and on real-world datasets. It probes the model's ability to navigate long-horizon routes, handle diverse weather and lighting conditions, and transfer synthetic pre-training to real-world driving scenarios without violating traffic rules.

Datasets

• CARLA Town13 — total ?; splits: val (-1), train (-1)
• Bench2Drive — total ?; splits: test (-1)
• Longest6 v2 — total ?; splits: test (-1)
• NAVSIM v1 — total ?; splits: navtrain (-1), navtest (-1)
• NAVSIM v2 — total ?; splits: navhard (-1)
• WOD-E2E — total ?; splits: validation (-1)

Metrics

• NDS (primary) — range: [0, 1]
  • Normalized Driving Score, defined as RC × I, where RC is Route Completion and I is the distance-normalized Infraction Score.
• Driving Score (DS) — range: [0, 1]
  • Standard CARLA metric combining route completion and infraction penalties. Can be counterintuitive on long routes due to exponential IS decay.
• Success Rate (SR) — range: [0, 1]
  • The fraction of routes completed with zero infractions.
• PDMS — range: [0, 1]
  • Predictive Driver Model Score aggregating collision avoidance, progress, time-to-collision, driving-area compliance, and comfort.
• EPDMS — range: [0, 1]
  • Extended PDMS used in NAVSIM v2, incorporating additional rule-based and sub-metrics with proximity-based weighting for synthetic start states.
• RFS — range: [0, 1]
  • Rater Feedback Score, a human-annotated metric assigning full expert credit within predefined trust regions and exponentially decaying the score otherwise.

Input / output format

Input: Multi-view camera images, LiDAR point clouds, and radar object-level features (or positional encodings when sensors are unavailable), along with historical vehicle states and discrete navigation commands.

Output: Continuous driving control commands (steering, throttle, brake) or a planned trajectory for the next time step.

Scoring recipe

def compute_nds(route_completion, infraction_score, distance):
    I = infraction_score / distance if distance > 0 else 0
    return route_completion * I

def compute_success_rate(routes):
    infraction_free = sum(1 for r in routes if r.infractions == 0)
    return infraction_free / len(routes)

def compute_pdms(trajectory, ground_truth, environment):
    # Aggregates collision avoidance, progress, TTC, drivable area compliance, comfort
    return weighted_sum([collision_avoidance, progress, ttc, area_compliance, comfort])

Common pitfalls

• Driving Score (DS) can be misleading on long routes because the exponential decay of Infraction Score (IS) may reward agents for stopping early rather than completing the route.
• Training on data from the validation town (e.g., Town13 Train) artificially inflates performance and hides the true generalization gap to novel environments.
• DS discounts errors on short routes, making Success Rate (SR) necessary to accurately assess policy robustness and infraction-free completion.

Evidence (verbatim from paper)

Metrics: We adopt the standard CARLA Leaderboard 2.0 metrics from Section 3.1. On long routes, however, the Driving Score (DS) can be counterintuitive: because the Infraction Score (IS) decays exponentially with each violation, agents are sometimes rewarded for stopping early rather than driving further [64]. We therefore report the Normalized Driving Score (NDS) on Town13, defined as RC × I, where I is distance-normalized version of IS [64]. For Bench2Drive, we additionally report Success Rate (SR), the fraction of infraction-free completions.

Citation

@misc{nguyen2025lead,
  title={LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving},
  author={Long Nguyen et al.},
  year={2025},
  note={arXiv:2512.20563}
}

• arXiv: 2512.20563