Wmabench Eval

wmabench-eval

Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation — Gao et al. (2025) (arXiv:2506.21876, 2025)

What this evaluates

Evaluates Vision-Language Models on atomic world modeling capabilities across perception (spatial, temporal, motion) and prediction (mechanistic simulation, transitive/compositional inference) tasks. It probes whether VLMs possess internal representations of physical causality, dynamics, and multi-step reasoning comparable to human intuition.

Datasets

• WM-ABench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of all evaluated instances. Calculated as (number of correct predictions) / (total number of parsed instances).

Input / output format

Input: Image or video frames paired with a text question/prompt. All models receive a fixed system prompt and are evaluated using greedy decoding.

Output: Free-text response that is subsequently parsed into a categorical label using a predefined template. Instances that fail parsing are excluded from scoring.

Scoring recipe

correct = 0
total = 0
for instance in dataset:
    output = model.generate(instance.image, instance.prompt, greedy=True)
    label = parse_template(output)
    if label is not None:
        total += 1
        if label == instance.ground_truth:
            correct += 1
return correct / total if total > 0 else 0.0

Common pitfalls

• Parsing failures are skipped rather than penalized, which may bias accuracy if certain models or tasks systematically fail to follow the output template.
• Greedy decoding is enforced across all models for consistency, potentially underestimating the performance of models that benefit from sampling or temperature tuning.
• Near-random performance on dynamic tasks (e.g., motion trajectory) reflects representation gaps rather than mere hallucination, requiring careful interpretation beyond raw scores.

Evidence (verbatim from paper)

We evaluate model performance by comparing the parsed labels from model outputs to the ground-truth labels, and skip the instances where model outputs failed to be parsed by our template. ... Qwen2-VL achieves the highest overall performance with an average accuracy of 67.7%.

Citation

@misc{gao2025wmabench,
  title={Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2506.21876}
}

• arXiv: 2506.21876