Navsim Pdm Eval

navsim-pdm-eval

Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillation — Li et al. (2025) (arXiv:2503.12820, 2025)

What this evaluates

Evaluates end-to-end autonomous driving planners on trajectory prediction and safety-critical behaviors. It measures compliance with traffic rules, drivable area boundaries, collision avoidance, and driving comfort over short-horizon scenarios.

Datasets

• NAVSIM — total 1328; splits: train (1192), test (136)

Metrics

• PDM Score (PDMS) (primary) — range: percent
  • PDMS = NC × DAC × (5×TTC + 2×C + 5×EP) / 12. Sub-metrics are percentages: No at-fault Collision (NC), Drivable Area Compliance (DAC), Time-to-Collision (TTC), Comfort (C), and Ego Progress (EP).
• Extended PDM Score (EPDMS) — range: percent
  • EPDMS = NC × DAC × DDC × TL × (5×TTC + 2×C + 5×EP + 5×LK + 5×EC) / 22. Adds Traffic Lights Compliance (TL), Driving Direction Compliance (DDC), Lane Keeping Ability (LK), and Extended Comfort (EC) to the base formula.

Input / output format

Input: Multi-view images (front, front-left, front-right) concatenated into 256×1024 resolution, 2 past frames, ego vehicle state (velocity, acceleration), and navigation commands (turning, lane changing, following).

Output: 40-waypoint trajectory over 4 seconds (10 Hz sampling), each waypoint defined by (x, y, heading) coordinates.

Scoring recipe

def compute_pdms(predictions, gold, ego_state):
    # Run NAVSIM benchmark simulator to extract sub-metrics
    # Returns NC, DAC, TTC, C, EP as percentages [0, 100]
    nc, dac, ttc, c, ep = run_pdm_eval(predictions, gold, ego_state)
    # Normalize to [0, 1] for formula multiplication
    nc, dac, ttc, c, ep = nc/100, dac/100, ttc/100, c/100, ep/100
    pdms = nc * dac * (5 * ttc + 2 * c + 5 * ep) / 12.0
    return pdms * 100  # Return as percentage to match reported table values

Common pitfalls

• The sub-metrics (NC, DAC, TTC, etc.) are computed by the NAVSIM benchmark simulator, not analytically from the trajectory alone; users must run the full simulation environment to get accurate values.
• The extended metrics require specific threshold configurations (e.g., τ_D = 0.5m for DDC/LK, τ_A = 0.7m/s² for EC) that must be set exactly as in the benchmark to reproduce scores.
• Comparing against LiDAR-based baselines without noting modality differences is misleading, as this method explicitly relies only on camera inputs and a lightweight ResNet34 backbone.

Evidence (verbatim from paper)

The dataset is split into two parts: Navtrain and Navtest, which respectively contain 1192 and 136 scenarios for training/validation and testing. Metrics. For NAVSIM dataset, we evaluate our models based on the PDM score (PDMS) and the Extended PDM Score (EPDMS), which can be formulated as follows: PDMS = NC × DAC × (5×TTC + 2×C + 5×EP)/12, EPDMS = NC × DAC × DDC × TL × (5×TTC + 2×C + 5×EP + 5×LK + 5×EC)/22 where sub-metrics NC, DAC, TTC, C, EP, DDC, TL, LK and EC correspond to the No at-fault Collision, Drivable Area Compliance, Time-to-Collision, Comfort, Ego Progress, Traffic Lights Compliance, Lane Keeping Ability and Extended Comfort.

Citation

@misc{li2025hydramdpp,
  title={Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2503.12820}
}

• arXiv: 2503.12820