Carla Leaderboard Eval

carla-leaderboard-eval

Self-driving cars: Are we there yet? — Atasever et al. (2025) (arXiv:2509.22754, 2025)

What this evaluates

Evaluates autonomous driving agents on their ability to navigate complex urban environments while balancing route completion, safety, and rule compliance. It probes how well models handle dynamic traffic interactions, edge-case scenarios, and physical constraints in a closed-loop simulation.

Datasets

• CARLA Leaderboard 2.0 scenarios — total ?; splits: dev-test (-1)
• CARLA 42 Routes — total 42; splits: test (42)
• Town05 — total 20; splits: Town05 Short (10), Town05 Long (10)

Metrics

• Driving Score (DS) (primary) — range: [0, 100]
  • DS_i = R_i × P_i, where R_i is the route completion percentage and P_i is the infraction penalty. Ranges from 0 to 100.
• Route Completion (RC) — range: [0, 100]
  • RC_i = 100 × (distance traversed / total route distance). Measures the percentage of the planned route successfully covered.
• Infraction Penalty (IP) — range: [0, 100]
  • IP_i = 100 × ∏_j p_j^{n_ij}, where n_ij is the count of infraction type j on route i, and p_j is the penalty coefficient. Higher values indicate fewer violations.

Input / output format

Input: Predefined routes specified via GPS coordinates and map waypoints, along with dynamically spawned traffic scenarios (e.g., obstacles, pedestrians, other vehicles) based on the NHTSA pre-crash typology.

Output: Continuous control commands (steering, acceleration, braking) or trajectory waypoints generated per timestep to navigate the route.

Scoring recipe

def evaluate_route(route, agent_trace):
    rc = 100 * (agent_trace.distance_traversed / route.total_distance)
    infractions = count_infractions(agent_trace.actions)
    ip = 100 * prod(p_j ** n_ij for j, n_ij in infractions)
    ds = rc * ip  # Paper formula; leaderboard typically normalizes to [0,100]
    return ds, rc, ip

Common pitfalls

• Domain mismatch between training data (Waymo Open Dataset) and evaluation simulator (CARLA) can cause prediction models to fail or behave conservatively.
• MPC-based planners may prioritize safety over progress, leading to artificially low route completion scores despite high infraction penalties.
• Evaluation requires closed-loop execution; open-loop or replay-based metrics do not capture compounding control errors or real-time interaction failures.

Evidence (verbatim from paper)

In the CARLA Leaderboard, agent performance is evaluated using three complementary metrics, each computed per route and then averaged to produce an overall score. ... Driving Score: DS_i = R_i P_i where R_i is the percentage of completion of and P_i is the infraction penalty of the i-th route. The driving score is the main metric of the leaderboard and ranges from 0 to 100.

Citation

@misc{atasever2025selfdriving,
  title={Self-driving cars: Are we there yet?},
  author={Atasever et al. (2025)},
  year={2025},
  note={arXiv:2509.22754}
}

• arXiv: 2509.22754