Ewmbench Eval

ewmbench-eval

EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models — Yue Hu et al. (2025) (arXiv:2505.09694, 2025)

What this evaluates

EWMBench evaluates embodied world models on their ability to generate videos that maintain static scene consistency, follow physically plausible motion trajectories, and align semantically with text instructions. It probes whether video generation models can produce action-consistent, task-grounded behaviors for robotic manipulation rather than just visually plausible but static or semantically drifting clips.

Datasets

• Agibot-World — total 100; splits: test (100); repo https://github.com/AgibotTech/EWMBench

Metrics

• Overall (primary) — range: [0, 1]
  • Normalized 0-1 score aggregating scene consistency, motion trajectory alignment (HSD, DYN, nDTW), and semantic alignment (Diversity, BLEU, CLIP, Logics). Higher values indicate better performance.
• SceneC — range: [0, 1]
  • Normalized score (0-1) measuring spatial layout and object presence preservation between generated and ground-truth videos.
• Motion_Avg — range: [0, 1]
  • Average of HSD, DYN, and nDTW scores (each 0-1), measuring trajectory alignment, spatial accuracy, and motion smoothness.
• Semantics_Avg — range: [0, 1]
  • Average of Diversity, BLEU, CLIP, and Logics scores (each 0-1), measuring instruction fidelity and task logic alignment.

Input / output format

Input: Image + text instruction (prompt) for Image-Text-to-Video generation.

Output: Generated video clip per episode.

Scoring recipe

scores = {}
for model in models:
    for episode in episodes:
        gt_video = load_ground_truth(episode)
        gen_video = generate_video(model, episode.image, episode.prompt)
        # Video-based MLLM evaluation
        scores[episode.id] = mllm_evaluate(gen_video, gt_video, episode.prompt)
    # Aggregate normalized scores (0-1) per dimension
    scores['SceneC'] = normalize(scores['scene_consistency'])
    scores['Motion_Avg'] = mean(normalize(scores['HSD']), normalize(scores['DYN']), normalize(scores['nDTW']))
    scores['Semantics_Avg'] = mean(normalize(scores['Diversity']), normalize(scores['BLEU']), normalize(scores['CLIP']), normalize(scores['Logics']))
    scores['Overall'] = scores['SceneC'] + scores['Motion_Avg'] + scores['Semantics_Avg']
return scores

Common pitfalls

• Static videos can score high in scene consistency but fail motion correctness; evaluators must check both dimensions independently.
• VBench metrics often fail to separate foreground/background and miss viewpoint changes, making them unreliable for embodied scenarios compared to EWMBench's SceneC.
• The benchmark currently only supports Image-Text-to-Video generation; action-conditioned models are explicitly excluded from this evaluation cycle.

Evidence (verbatim from paper)

We evaluate models across dimensions using normalized scores between 0 and 1, where higher values indicate better performance. Results in Table 2 show that domain-adapted models, such as EnerVerse and LTX_FT, consistently outperform commercial models (e.g., Kling, Hailuo) and open-source models (e.g., COSMOS, OpenSora, LTX). This highlights the effectiveness of domain-specific fine-tuning in capturing motion dynamics and task semantics. Notably, EnerVerse and Kling demonstrate strong semantic alignment, reflecting a solid understanding of task logic.

Citation

@misc{hu2025ewmbench,
  title={EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models},
  author={Yue Hu et al. (2025)},
  year={2025},
  note={arXiv:2505.09694}
}

• arXiv: 2505.09694