Wowbench Eval

wowbench-eval

WoW: Towards a World omniscient World model Through Embodied Interaction — Chi et al. (2025) (arXiv:2509.22642, 2025)

What this evaluates

Evaluates embodied world models on conditional video generation from an initial image and text instruction. It probes instruction understanding, long-horizon planning, physical/causal reasoning, and temporal consistency in robotic interaction scenarios.

Datasets

• WoWBench — total ?; splits: test (-1)

Metrics

• FVD — range: other
  • Fréchet Video Distance measuring distributional difference between generated and real video feature embeddings.
• Mask-guided Regional Consistency — range: [0, 1]
  • Cosine similarity of region-specific embeddings (DINOv3) computed separately for robot arm, manipulated object, and background across frames.
• Sequence Match Score — range: percent
  • LLM-evaluated score measuring the correct ordering of actions in the generated video relative to the instruction.
• Execution Quality Score — range: other
  • LLM-evaluated score on a 1–5 scale assessing how well the generated video fulfills the instruction.
• Trajectory Consistency — range: other
  • Combines Mean Euclidean Distance (MED), Dynamic Time Warping (DTW), and Fréchet Distance to compare tracked end-effector/object trajectories between generated and ground-truth videos.
• Physical Common Sense Score — range: other
  • 1–5 scale rating across six dimensions (object interaction, properties, temporal consistency, lighting, fluid dynamics, local anomalies) via a fine-tuned Qwen-2.5-VL evaluator.
• Planning Score ($S_{plan}$) (primary) — range: [0, 1]
  • $S_{plan} = (0.5 \times R_k + 0.5 \times R_s) \times P_k$, where $R_k$ is key-step recall, $R_s$ is sequential consistency, and $P_k$ is key-step precision, computed by comparing DAGs of predicted vs. ground-truth action plans.
• Overall Benchmark Score (primary) — range: other
  • Weighted arithmetic mean of per-metric desirability scores $s_{i,m} \in (0,100)$, where raw metrics are pre-scaled to [0,1] using absolute anchors, transformed via monotone mappings, and aggregated by group and overall.

Input / output format

Input: Initial image and a natural language instruction specifying the desired action/task.

Output: A generated video sequence depicting the execution of the instruction from the initial state.

Scoring recipe

# 1. Compute raw metrics per model i and metric m: x_i,m
# 2. Pre-scale to [0,1] using absolute anchors L_m, U_m
if higher_is_better:
    x_hat = clip(x_i,m, L_m, U_m)
    x_hat = (x_hat - L_m) / (U_m - L_m)
else:
    x_hat = 1 - (clip(x_i,m, L_m, U_m) - L_m) / (U_m - L_m)
# 3. Apply monotone mapping f_m (e.g., power, logit, tanh) and scale to (0,100)
s_i,m = 100 * f_m(x_hat; theta_m)
# 4. Group averaging (uniform)
G_i,g = mean(s_i,m for m in group_g)
# 5. Overall aggregation (weighted mean)
O_i = sum(W_g * G_i,g for g in groups) / sum(W_g for g in groups)
return O_i

Common pitfalls

• Relies on LLM/VLM evaluators (GPT-4o, Gemini, Qwen-2.5-VL) for instruction, planning, and physical common sense scores, which may introduce systematic bias or inconsistency compared to human judgment.
• The overall benchmark score uses dataset-tuned monotone mappings and absolute anchors, which can compress or distort raw metric differences and make cross-benchmark comparisons difficult.
• Trajectory consistency metrics depend on SAM2 tracking, which can degrade significantly under heavy occlusion or rapid motion, skewing physical reasoning scores.

Evidence (verbatim from paper)

We first parse the natural language instruction and ground-truth video into a ground-truth plan DAG, where nodes are atomic actions and edges represent dependencies. This representation flexibly handles non-unique but valid action orderings. We then compare the model-generated plan (which also uses the same approach to infer from the video) to the ground-truth DAG using three scores: Key-step Recall, Sequential Consistency, and Key-step Precision. The final planning score $S_{plan}$ integrates these aspects to reward both completeness and correctness: $S_{plan} = (0.5 \times R_k + 0.5 \times R_s) \times P_k$

Citation

@misc{chi2025wow,
  title={WoW: Towards a World omniscient World model Through Embodied Interaction},
  author={Chi et al. (2025)},
  year={2025},
  note={arXiv:2509.22642}
}

• arXiv: 2509.22642