Worldmark Eval

worldmark-eval

WorldMark: A Unified Benchmark Suite for Interactive Video World Models — Xu et al. (2026) (arXiv:2604.21686, 2026)

What this evaluates

This benchmark evaluates interactive Image-to-Video world models by measuring their ability to generate temporally coherent videos in response to standardized action commands. It probes three core capabilities: visual fidelity, precise camera/object control alignment, and long-horizon world consistency across different perspectives and visual styles.

Datasets

• WorldMark Image Suite — total 50; splits: First-Person Real (-1), First-Person Stylized (-1), Third-Person Real (-1), Third-Person Stylized (-1)

Metrics

• Aesthetic Quality (primary) — range: other
  • Automated score measuring visual appeal and artistic merit of generated frames. Scale unspecified in text.
• Imaging Quality — range: other
  • Automated score measuring technical rendering fidelity and clarity of generated frames.
• Translation Error — range: other
  • Geometric deviation between intended and actual camera/object translation in the generated video.
• Rotation Error — range: other
  • Angular deviation between intended and actual camera/object rotation in the generated video.
• Reprojection Error — range: other
  • Error in projecting 3D scene structure back to 2D frames, measuring spatial consistency.
• State Consistency — range: other
  • Score measuring temporal stability of object states and physics across the video sequence.
• Content Consistency — range: other
  • Score measuring preservation of scene elements and objects without hallucination or disappearance.
• Style Consistency — range: other
  • Score measuring maintenance of the original visual style and artistic tone throughout the generation.

Input / output format

Input: A reference image (first- or third-person view) paired with a sequence of 5 standardized WASD-style action commands, translated via a unified interface into the model's native control format (e.g., captions, pose vectors, or gamepad signals).

Output: A generated video sequence corresponding to the input action sequence, rendered in the specified perspective and style.

Scoring recipe

def evaluate_model(model, test_suite):
    metrics = {'Aesthetic': [], 'Imaging': [], 'TransErr': [], 'RotErr': [], 'ReprojErr': [], 'StateCon': [], 'ContentCon': [], 'StyleCon': []}
    for img, actions in test_suite:
        video = model.generate(img, actions)
        metrics['Aesthetic'].append(compute_aesthetic_score(video))
        metrics['Imaging'].append(compute_imaging_score(video))
        metrics['TransErr'].append(compute_translation_error(video, actions))
        metrics['RotErr'].append(compute_rotation_error(video, actions))
        metrics['ReprojErr'].append(compute_reprojection_error(video))
        metrics['StateCon'].append(compute_state_consistency(video))
        metrics['ContentCon'].append(compute_content_consistency(video))
        metrics['StyleCon'].append(compute_style_consistency(video))
    return {k: mean(v) for k, v in metrics.items()}

Common pitfalls

• Models require a unified action-mapping layer to translate standardized commands into native interfaces; skipping this step breaks cross-model comparability.
• Third-person evaluation is only valid for models natively supporting that perspective (Matrix-Game 2.0, HY-World 1.5, Genie 3); forcing unsupported models yields invalid results.
• High visual quality does not guarantee world consistency or control alignment; these dimensions are largely uncorrelated and must be evaluated independently.

Evidence (verbatim from paper)

Regarding control alignment, the results indicate a divergence in model capabilities. HY-Game exhibits robust translation control, achieving the lowest Translation Error in both Real (0.159) and Stylized (0.116) scenarios, as well as the lowest Rotation Error (0.932) in the Stylized setting.

Citation

@misc{xu2026worldmark,
  title={WorldMark: A Unified Benchmark Suite for Interactive Video World Models},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2604.21686}
}

• arXiv: 2604.21686