Carla No Crash Eval

carla-no-crash-eval

Found-RL: foundation model-enhanced reinforcement learning for autonomous driving — Qu et al. (2026) (arXiv:2602.10458, 2026)

What this evaluates

Evaluates autonomous driving agents' robustness and generalization in a simulated urban environment, specifically testing performance on familiar and unseen town layouts.

Datasets

• CARLA NoCrash benchmark — total ?; splits: train (-1), test (-1)

Metrics

• Driving Score (primary) — range: [0, 1]
  • Product of route completion percentage and an infraction penalty discount factor.
• Success Rate — range: [0, 1]
  • Fraction of episodes successfully completed without triggering terminal failure conditions.
• Route Completion — range: [0, 100] percent
  • Percentage of the reference route distance successfully traversed.
• Infraction Penalty — range: [0, 1]
  • Discount factor aggregating penalties for traffic rule violations (e.g., red lights, collisions).
• Collisions of Vehicle — range: other
  • Number of vehicle collisions normalized by completed route length.
• Collisions of Pedestrian — range: other
  • Number of pedestrian collisions normalized by completed route length.
• Red Light — range: other
  • Number of red light violations over the completed route length.
• Return — range: other
  • Accumulated rewards over the entire episode.
• Icell — range: other
  • Estimated battery cell current based on a single-cell equivalent circuit model.
• Fuel Rate — range: other
  • Instantaneous fuel consumption derived from a longitudinal road-load model.
• Speed — range: other
  • Average driving speed over the episode.

Input / output format

Input: BEV image (192x192x3) and text prompt for VLMs; BEV masks (96x96x15) and compact state vector (vehicle motion, last-step controls, traffic context) for RL agents.

Output: Continuous 3D action (throttle, steer, brake) for VLMs; continuous 2D action (signed longitudinal, steer) for RL agents.

Scoring recipe

Driving Score = Route Completion * Infraction Penalty.
Episode terminates on collision, route deviation, traffic violation, or completion.
Metrics are averaged over 3 independent runs with different random seeds.

Common pitfalls

• Evaluation uses a stricter termination logic than the original ROACH/Leaderboard protocol, including tighter route adherence checks and additional failure triggers.
• Physics for BEV-invisible static objects are disabled to prevent unobservable collision terminations that disrupt credit assignment.

Evidence (verbatim from paper)

To comprehensively evaluate the performance of the driving agents, we use a set of metrics organized into four primary categories: Comprehensive, Route, Energy, and Safety. The Comprehensive metrics include Return, which represents the accumulated rewards over the entire episode; Driving Score, defined as the product of route completion and the infraction penalty; and Infraction Penalty, a discount factor aggregating all triggered infractions.

Citation

@misc{qu2026foundrl,
  title={Found-RL: foundation model-enhanced reinforcement learning for autonomous driving},
  author={Qu et al. (2026)},
  year={2026},
  note={arXiv:2602.10458}
}

• arXiv: 2602.10458