navsim-epdms-eval
SimScale: Learning to Drive via Real-World Simulation at Scale — Haochen Tian et al. (2025) (arXiv:2511.23369, 2025)
What this evaluates
Evaluates closed-loop end-to-end autonomous driving performance in safety-critical and diverse real-world scenarios. It measures collision avoidance, rule compliance, progress, and comfort under reactive traffic conditions.
Datasets
- navhard — total 4408; splits: stage1 (244), stage2 (4164)
- navtest — total 12146; splits: test (12146)
Metrics
EPDMS (primary) — range: percent
- EPDMS = (∏ S_m for m in M_pen) * (∑ w_m S_m / ∑ w_m for m in M_avg). Penalty terms (M_pen) include No-at-fault Collisions (NC), Drivable Area Compliance (DAC), Driving Direction Compliance (DDC), and Traffic Light Compliance (TLC). Weighted average terms (M_avg) include Time-to-Collision (TTC), Ego Progress (EP), Lane Keeping (LK), History Comfort (HC), and extended comfort (EC). Weights w_m are defined by the NAVSIM benchmark.
Input / output format
Input: Multi-view camera images (unified to 2048×512 resolution) without LiDAR inputs. The model processes these to generate closed-loop driving trajectories.
Output: Closed-loop driving actions/trajectories evaluated over a simulation rollout.
Scoring recipe
def compute_epdms(sub_metrics):
pen_terms = ['NC', 'DAC', 'DDC', 'TLC']
avg_terms = ['TTC', 'EP', 'LK', 'HC', 'EC']
weights = {m: 1.0 for m in avg_terms} # weights per NAVSIM spec
penalty_score = 1.0
for m in pen_terms:
penalty_score *= sub_metrics[m]
avg_score = sum(weights[m] * sub_metrics[m] for m in avg_terms) / sum(weights.values())
return penalty_score * avg_score
Common pitfalls
- navhard uses a two-stage aggregation where Stage 2 scenarios are synthetically generated from Stage 1 failures.
- Penalties are excluded if the human expert driver also fails in that scenario.
- Regression-based planners may degrade under high simulation-to-real ratios due to multi-modal supervision, unlike diffusion models.
Evidence (verbatim from paper)
The two benchmarks share a rule-based planning metric, EPDMS [44], with several sub-metrics: EPDMS = (∏ S_m for m in M_pen) * (∑ w_m S_m / ∑ w_m for m in M_avg), where S_m is the sub-metric: penalty terms set M_pen includes No-at-fault Collisions (NC), Drivable Area Compliance (DAC), Driving Direction Compliance (DDC), and Traffic Light Compliance (TLC); weighted average terms set M_avg includes Time-to-Collision (TTC), Ego Progress (EP), Lane Keeping (LK), History Comfort (HC), and extended comfort (EC).
Citation
@misc{tian2025simscale,
title={SimScale: Learning to Drive via Real-World Simulation at Scale},
author={Haochen Tian et al. (2025)},
year={2025},
note={arXiv:2511.23369}
}
1---2name: navsim-epdms-eval3description: Evaluates closed-loop end-to-end autonomous driving performance in safety-critical and diverse real-world scenarios. It measures collision avoidance, rule compliance, progress, and comfort under reactive traffic conditions. Use when the user wants to benchmark on navhard, navtest, or asks about evaluating this task. Reports EPDMS.4---56# navsim-epdms-eval78> SimScale: Learning to Drive via Real-World Simulation at Scale — Haochen Tian et al. (2025) (arXiv:2511.23369, 2025)910## What this evaluates1112Evaluates closed-loop end-to-end autonomous driving performance in safety-critical and diverse real-world scenarios. It measures collision avoidance, rule compliance, progress, and comfort under reactive traffic conditions.1314## Datasets1516- **navhard** — total 4408; splits: stage1 (244), stage2 (4164)17- **navtest** — total 12146; splits: test (12146)1819## Metrics2021- `EPDMS` **(primary)** — range: percent22 - EPDMS = (∏ S_m for m in M_pen) * (∑ w_m S_m / ∑ w_m for m in M_avg). Penalty terms (M_pen) include No-at-fault Collisions (NC), Drivable Area Compliance (DAC), Driving Direction Compliance (DDC), and Traffic Light Compliance (TLC). Weighted average terms (M_avg) include Time-to-Collision (TTC), Ego Progress (EP), Lane Keeping (LK), History Comfort (HC), and extended comfort (EC). Weights w_m are defined by the NAVSIM benchmark.2324## Input / output format2526**Input**: Multi-view camera images (unified to 2048×512 resolution) without LiDAR inputs. The model processes these to generate closed-loop driving trajectories.2728**Output**: Closed-loop driving actions/trajectories evaluated over a simulation rollout.2930## Scoring recipe3132```python33def compute_epdms(sub_metrics):34 pen_terms = ['NC', 'DAC', 'DDC', 'TLC']35 avg_terms = ['TTC', 'EP', 'LK', 'HC', 'EC']36 weights = {m: 1.0 for m in avg_terms} # weights per NAVSIM spec37 penalty_score = 1.038 for m in pen_terms:39 penalty_score *= sub_metrics[m]40 avg_score = sum(weights[m] * sub_metrics[m] for m in avg_terms) / sum(weights.values())41 return penalty_score * avg_score42```4344## Common pitfalls4546- navhard uses a two-stage aggregation where Stage 2 scenarios are synthetically generated from Stage 1 failures.47- Penalties are excluded if the human expert driver also fails in that scenario.48- Regression-based planners may degrade under high simulation-to-real ratios due to multi-modal supervision, unlike diffusion models.4950## Evidence (verbatim from paper)5152> The two benchmarks share a rule-based planning metric, EPDMS [44], with several sub-metrics: EPDMS = (∏ S_m for m in M_pen) * (∑ w_m S_m / ∑ w_m for m in M_avg), where S_m is the sub-metric: penalty terms set M_pen includes No-at-fault Collisions (NC), Drivable Area Compliance (DAC), Driving Direction Compliance (DDC), and Traffic Light Compliance (TLC); weighted average terms set M_avg includes Time-to-Collision (TTC), Ego Progress (EP), Lane Keeping (LK), History Comfort (HC), and extended comfort (EC).5354## Citation5556```bibtex57@misc{tian2025simscale,58 title={SimScale: Learning to Drive via Real-World Simulation at Scale},59 author={Haochen Tian et al. (2025)},60 year={2025},61 note={arXiv:2511.23369}62}63```6465- arXiv: 2511.23369
qhjqhj00