Omniworld Game Eval

omniworld-game-eval

OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling — Zhou et al. (2025) (arXiv:2509.12201, 2025)

What this evaluates

Evaluates 3D geometric foundation models on monocular and video depth estimation, and tests camera-controlled video generation models on their ability to follow camera trajectories while maintaining video quality across diverse, dynamic environments.

Datasets

• OmniWorld-Game — total ?; splits: test (-1); repo https://github.com/yangzhou24/OmniWorld

Metrics

• Abs Rel — range: other
  • Absolute relative error between predicted and ground truth depth. Lower is better.
• δ<1.25 — range: percent
  • Percentage of pixels where the ratio of predicted to ground truth depth is within 1.25. Higher is better.
• TransErr — range: other
  • Translation error between predicted and ground truth camera poses. Lower is better.
• RotErr — range: other
  • Rotation error between predicted and ground truth camera poses. Lower is better.
• CamMC — range: other
  • Camera Motion Consistency metric quantifying adherence to camera commands. Lower is better.
• FVD (primary) — range: other
  • Fréchet Video Distance measuring perceptual realism between generated and real video distributions. Lower is better.

Input / output format

Input: Monocular depth: single RGB image. Video depth: video sequence. Video generation: text prompt or reference image plus camera trajectory parameters.

Output: Depth maps or video depth sequences for geometric tasks; generated video frames for video generation tasks.

Scoring recipe

abs_rel = np.mean(np.abs(pred_depth - gt_depth) / gt_depth)
delta1 = np.mean((pred_depth / gt_depth < 1.25) & (gt_depth / pred_depth < 1.25))
trans_err = np.linalg.norm(pred_cam_t - gt_cam_t)
rot_err = np.linalg.norm(R_log(pred_cam_R) - R_log(gt_cam_R))
cam_mc = compute_camera_motion_consistency(pred_traj, gt_traj)
fvd = frechet_video_distance(real_videos, generated_videos)

Common pitfalls

• Models often fail to maintain temporal consistency over long sequences (up to 384 frames).
• Scale-and-shift alignment drastically changes depth metric scores compared to scale-only alignment.
• Video generation models struggle to simultaneously achieve high camera control accuracy and perceptual video quality.

Evidence (verbatim from paper)

Following CAMI2V (Zheng et al., [2024]), metrics include Camera Parameter Metrics (RotError, TransError, and CamMC) to quantify adherence to camera commands, and Fréchet Video Distance (FVD) (Unterthiner et al., [2018]) to assess perceptual realism.

Citation

@misc{zhou2025omniworld,
  title={OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2509.12201}
}

• arXiv: 2509.12201