Cow Bench Eval

cow-bench-eval

The Trinity of Consistency as a Defining Principle for General World Models — Wei et al. (2026) (arXiv:2602.23152, 2026)

What this evaluates

Evaluates modal, spatial, and temporal consistency in general world models through 18 sub-tasks across six task categories. It uses human-designed checklists to verify fine-grained physical laws, causal reasoning, and cross-modal alignment, moving beyond perceptual metrics to hard verification.

Datasets

• CoW-Bench — total ?; splits: test (-1)

Metrics

• checklist_score (primary) — range: percent
  • Aggregated pass rate across five human-designed checklists per sub-task. Each checklist verifies a specific physical, logical, or cross-modal constraint, with scores normalized to a 0-100% scale.

Input / output format

Input: Text prompts, images, and video sequences designed to test modal, spatial, and temporal consistency, including counterfactual interventions and multi-frame reasoning tasks.

Output: Generated images/videos or reasoning traces, evaluated against five checklist criteria per sub-task to verify physical laws, causal chains, and cross-modal alignment.

Scoring recipe

def evaluate_cow_bench(predictions, gold, checklists):
    total_checks = 0
    passed_checks = 0
    for task in predictions:
        for checklist_item in checklists[task.subtask]:
            total_checks += 1
            if verify_physical_logic(task.output, checklist_item):
                passed_checks += 1
    return (passed_checks / total_checks) * 100

Common pitfalls

• Relies on human checklists rather than automated differentiable metrics, making large-scale evaluation costly and subjective.
• Current benchmarks it critiques (e.g., TiViBench, V-ReasonBench) over-rely on VLM-as-a-judge, which fails on fine-grained physical attributes like friction or momentum conservation.
• Long-range generation errors (butterfly effect) are masked by short-sequence testing protocols common in prior work.

Evidence (verbatim from paper)

Each sub-task is paired with five carefully designed human checklists, yielding a comprehensive, task-driven protocol with fine-grained criteria to pinpoint complementary failure modes and enable more precise, interpretable quantification.

Citation

@misc{wei2026trinity,
  title={The Trinity of Consistency as a Defining Principle for General World Models},
  author={Wei et al. (2026)},
  year={2026},
  note={arXiv:2602.23152}
}

• arXiv: 2602.23152