Drivinggen Eval

drivinggen-eval

DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving — Zhou et al. (2026) (arXiv:2601.01528, 2026)

What this evaluates

Evaluates generative video world models for autonomous driving by jointly assessing visual realism, trajectory plausibility, temporal and agent-level consistency, and ego-conditioned motion controllability over a 100-frame prediction horizon. It benchmarks both general-purpose and driving-specific models to reveal trade-offs between photorealism and physical motion fidelity.

Datasets

• DrivingGen — total ?; splits: test (-1)

Metrics

• Avg. Rank (primary) — range: integer rank
  • Models are ranked on each individual metric (1 = best). The average of these ranks across all reported metrics serves as a quick summary score.
• FVD — range: arbitrary (lower is better)
  • Fréchet Video Distance; measures distribution-level similarity between generated and real video frames.
• FTD — range: arbitrary (lower is better)
  • Fréchet Trajectory Distance; measures distribution-level similarity of predicted trajectories.
• ADE — range: arbitrary (lower is better)
  • Average Displacement Error; measures trajectory alignment under ego-conditioning.
• DTW — range: arbitrary (lower is better)
  • Dynamic Time Warping distance; measures temporal alignment and adherence to commanded paths.

Input / output format

Input: Initial driving scene video frame(s) and ego-vehicle trajectory commands/conditions.

Output: A generated video sequence of 100 frames conditioned on the input trajectory.

Scoring recipe

def evaluate(predictions, gold):
    # predictions: generated 100-frame videos
    # gold: ground truth videos & trajectory conditions
    scores = {}
    scores['FVD'] = frechet_video_distance(predictions, gold)
    scores['ADE'] = average_displacement_error(predictions, gold)
    scores['DTW'] = dtw_distance(predictions, gold)
    # Compute remaining metrics (FTD, SubjectiveQuality, VideoConsist, etc.)
    # Rank models per metric (1 = best)
    ranks = {m: rank_models(scores[m]) for m in scores}
    scores['Avg. Rank'] = mean([ranks[m] for m in ranks])
    return scores

Common pitfalls

• Relying solely on distribution-level metrics like FVD/FTD can mask physically implausible behaviors (e.g., stop-go jitter, agent disappearance).
• High objective quality scores (e.g., low flicker) may coexist with poor subjective quality or unstable agent behavior.
• Treating the average rank as a definitive score rather than a quick summary metric.

Evidence (verbatim from paper)

Existing benchmarks often rely solely on distribution-level metrics such as FVD to evaluate generated driving videos. While useful for assessing overall distribution similarity, good FVD/FTD alone does not necessarily imply plausible driving—videos can appear distribution-close yet exhibit stop–go jitter, identity drift, or non-physical disappearances.

Citation

@misc{zhou2026drivinggen,
  title={DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving},
  author={Zhou et al. (2026)},
  year={2026},
  note={arXiv:2601.01528}
}

• arXiv: 2601.01528