Recogdrive Eval

recogdrive-eval

ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving — Yongkang Li et al. (arXiv:2506.08052, 2025)

What this evaluates

Evaluates an end-to-end autonomous driving agent's ability to generate safe, comfortable, and efficient driving trajectories using only camera inputs. It probes the model's closed-loop planning capabilities, safety-critical scenario handling, and visual reasoning in complex urban environments.

Datasets

• NAVSIM — total 1328; splits: train (1192), test (136)
• Bench2Drive — total 220; splits: test (220)

Metrics

• PDMS (primary) — range: [0, 100]
  • Planning Driving Metric Score, a composite closed-loop metric from the nuPlan/NAVSIM protocol that aggregates safety, comfort, efficiency, and progress scores into a single normalized value.
• NC — range: percent
  • No-Collision rate, measuring the percentage of simulation steps without collisions.
• DAC — range: percent
  • Drivable Area Compliance, measuring the percentage of time the vehicle stays within drivable boundaries.
• TTC — range: percent
  • Time-to-Collision metric, evaluating proximity to potential collisions over the trajectory.
• Comf — range: percent
  • Comfort score, penalizing harsh acceleration, braking, and steering maneuvers.
• EP — range: percent
  • Efficiency Progress, measuring how much of the route is completed relative to a baseline.
• DS — range: percent
  • Driving Score on Bench2Drive, a CARLA leaderboard metric combining efficiency, comfort, and success rate.
• DriveBench Avg — range: percent
  • Average score across perception, prediction, planning, and behavior sub-tasks on the DriveLM/DriveBench VQA benchmarks.

Input / output format

Input: Camera images processed via dynamic resolution preprocessing, along with standard driving environment observations (ego state, map, traffic participants) implicit in the NAVSIM/Bench2Drive protocols.

Output: Continuous, feasible driving trajectories generated by a diffusion planner.

Scoring recipe

def evaluate_closed_loop(model, dataset, sim_env):
    scores = []
    for scene in dataset:
        traj = model.predict(scene.images, scene.ego_state)
        sim_result = sim_env.run(traj)
        nc = sim_result.no_collision_rate
        dac = sim_result.drivable_area_compliance
        ttc = sim_result.time_to_collision
        comf = sim_result.comfort_score
        ep = sim_result.efficiency_progress
        pdms = composite_metric(nc, dac, ttc, comf, ep)
        scores.append(pdms)
    return mean(scores)

Common pitfalls

• The paper explicitly uses only camera input, unlike many baselines that use LiDAR; failing to match input modality leads to unfair comparisons.
• Evaluation is strictly closed-loop (simulated driving), not open-loop trajectory matching; metrics like PDMS depend on the agent's continuous interaction with the environment.
• Bench2Drive uses CARLA's leaderboard protocol with specific safety-critical scenarios; results are not directly comparable to open-loop or different closed-loop benchmarks.

Evidence (verbatim from paper)

We evaluate primarily on two challenging benchmarks: NAVSIM(Dauner et al., [2025]) and Bench2Drive(Jia et al., [2024]). NAVSIM is a planning-oriented autonomous driving dataset built on OpenScene*(Contributors, [2023]), a redistribution of nuPlan(Caesar et al., [2021])*. The dataset is split into navtrain (1,192 training scenes) and navtest (136 evaluation scenes). Bench2Drive is a CARLA-based benchmark composed of 220 short routes, each containing a distinct, safety-critical scenario. ReCogDrive achieves a PDMS of 90.8, establishing a new state-of-the-art.

Citation

@misc{li2025recogdrive,
  title={ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving},
  author={Yongkang Li et al.},
  year={2025},
  note={arXiv:2506.08052}
}

• arXiv: 2506.08052