Worldlens Eval

worldlens-eval

WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World — Liang et al. (2025) (arXiv:2512.10958, 2025)

What this evaluates

Evaluates driving world models across five dimensions: generation quality, 3D/4D reconstruction coherence, action-following capability in closed-loop simulation, downstream perception task utility, and alignment with human preference. It probes geometric consistency, physical plausibility, and functional reliability of synthesized driving scenes.

Datasets

• WorldLens — total ?; splits: test (-1); repo https://github.com/worldbench/WorldLens

Metrics

• FID (Subject Fidelity) — range: [0, inf)
  • Fréchet Inception Distance measuring the distance between feature distributions of real and generated driving frames. Lower is better.
• Perceptual Discrepancy — range: [0, 1] or dB
  • Likely LPIPS or PSNR-based metric quantifying visual similarity between generated and ground-truth frames. Lower is better.
• Geometric Discrepancy / Photometric Error — range: [0, inf)
  • Errors computed after reconstructing generated videos into 4D Gaussian Fields. Measures floaters, structural instability, and pixel-level reconstruction loss.
• Route Completion (%) (primary) — range: [0, 100] percent
  • Percentage of simulation episodes where a closed-loop planner successfully navigates the route without collisions or off-road drift.
• Human Preference Score — range: [1, 10]
  • Subjective ratings on a 1–10 scale across World Realism, Physical Plausibility, 3D & 4D Consistency, and Behavioral Safety.

Input / output format

Input: Synthesized video sequences or frame-by-frame predictions from driving world models, typically conditioned on ground-truth frames, ego-vehicle states, or action commands.

Output: Quantitative metric scores per evaluation aspect, plus human preference ratings on a 1–10 scale and textual rationales for the WorldLens-Agent auto-evaluator.

Scoring recipe

def compute_metrics(predictions, gold):
    scores = {}
    scores['FID'] = frechet_inception_distance(predictions, gold)
    scores['Perceptual'] = perceptual_similarity(predictions, gold)
    scores['Geo_Error'] = reconstruct_4d_gaussians(predictions).compute_error(gold)
    scores['Route_Completion'] = closed_loop_planner(predictions).success_rate()
    scores['Human_Pref'] = human_annotators.score(predictions, scale=10)
    return scores

Common pitfalls

• Relying solely on perceptual metrics (e.g., FID/LPIPS) without verifying geometric or physical consistency, as texture-rich models often violate physics.
• Evaluating only in open-loop settings; closed-loop simulation exposes severe planner failures (collisions, off-road drift) that open-loop metrics miss.
• Assuming high perceptual quality guarantees downstream task usability; multi-domain training can actually hinder task-specific adaptation.

Evidence (verbatim from paper)

As summarized in Table [1], all existing models remain notably below the ‘Empirical Max’, indicating substantial room for improving the visual and temporal realism of driving world models. Although DiST-4D achieves the lowest Perceptual Discrepancy, it underperforms OpenDWM in Subject Fidelity and View Consistency, demonstrating that perceptual metrics alone are insufficient for assessing physically coherent scene generation.

Citation

@misc{liang2025worldlens,
  title={WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World},
  author={Liang et al. (2025)},
  year={2025},
  note={arXiv:2512.10958}
}

• arXiv: 2512.10958