Pseudo Simulation Eval

pseudo-simulation-eval

Pseudo-Simulation for Autonomous Driving — Cao et al. (2025) (arXiv:2506.04218, 2025)

What this evaluates

This evaluation probes the closed-loop planning robustness and causal reasoning of autonomous vehicle controllers by measuring their ability to handle compounding errors and distribution shifts. It combines real-world driving observations with pseudo-synthetic future scenarios generated via neural rendering to approximate interactive simulation without requiring a full physics engine.

Datasets

• nuPlan (navhard subset) — total 5912; splits: Stage 1 (450), Stage 2 (5462); repo https://github.com/autonomousvision/navsim

Metrics

• EPDMS (primary) — range: [0, 1]
  • Composite planning score aggregating binary/continuous subscores (NC, DAC, DDC, TLC, EP, TTC, LK, HC, EC) using multiplicative aggregation across two 4-second stages to approximate an 8-second closed-loop score.
• CLS — range: [0, 1]
  • Closed-loop score computed by the nuPlan simulator for fully reactive rollouts with privileged perception and HD map access.
• mIoU — range: [0, 1]
  • Mean Intersection over Union for Bird’s Eye View segmentation over drivable area, walkway, and vehicle classes.
• LPIPS — range: [0, 1]
  • Learned perceptual image patch similarity measuring novel view synthesis fidelity against ground-truth frames.

Input / output format

Input: Per instance, the model receives a sequence of real-world sensor observations (Stage 1) and pseudo-synthetic observations generated via 3D Gaussian Splatting (Stage 2), including BEV images, HD maps, and privileged ground-truth perception data for privileged planners.

Output: A planned trajectory or control actions over a 4-second horizon per stage (total 8 seconds), typically at 10Hz frequency.

Scoring recipe

def compute_epdms(subscores_s1, subscores_s2):
    # subscores: NC, DAC, DDC, TLC, EP, TTC, LK, HC, EC
    # Multiplicative aggregation across stages
    score_s1 = 1.0
    score_s2 = 1.0
    for metric in subscores_s1:
        score_s1 *= metric
    for metric in subscores_s2:
        score_s2 *= metric
    # Hybrid strategy: multiply penalty metrics, average others
    # Paper defaults to multiplicative aggregation for binary-heavy subscores
    epdms = score_s1 * score_s2
    return epdms

Common pitfalls

• Assuming EPDMS is a simple arithmetic mean; the paper explicitly uses multiplicative aggregation to better handle binary penalty metrics.
• Comparing EPDMS directly to nuPlan's CLS without accounting for missing subscores (TLC, LK, EC) in the closed-loop simulator, which biases correlation analysis.
• Treating Stage 2 synthetic observations as independent test cases rather than distribution-shifted variants of Stage 1 real observations.

Evidence (verbatim from paper)

We measure the alignment between EPDMS and nuPlan’s closed-loop score (CLS) using Pearson’s linear (r) and Spearman’s rank (ρ) correlation coefficients, as well as the coefficient of determination (R2). Since R2 is calculated by fitting a linear model between EPDMS and CLS, it is equivalent to the square of Pearson’s correlation coefficient here (R2=r2). This assumes that an ideal pseudo-simulation metric should show a linear relationship with closed-loop scores, requiring no adjustments for scale or bias.

Citation

@misc{cao2025pseudosimulation,
  title={Pseudo-Simulation for Autonomous Driving},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2506.04218}
}

• arXiv: 2506.04218