Omnispatial Eval

omnispatial-eval

OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models — Jia et al. (2025) (arXiv:2506.03135, 2025)

What this evaluates

This benchmark evaluates vision-language models on advanced spatial reasoning capabilities, specifically probing dynamic reasoning, complex spatial logic, spatial interaction, and perspective-taking. It measures how well models understand and manipulate spatial relationships, temporal changes, and viewpoint shifts beyond basic object recognition.

Datasets

• OmniSpatial — total 8400; splits: train (6900), test (-1); repo https://github.com/qizekun/OmniSpatial

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. For standard models, correctness is determined via direct answer matching, regular-expression parsing, JSON parsing, or LLM-as-a-Judge. For reasoning models with unstructured chain-of-thought outputs, an external judge (GPT-4.1-mini) compares the final answer against ground truth.

Input / output format

Input: An image paired with a multiple-choice question requiring spatial reasoning.

Output: A selected option (e.g., A, B, C, D) and optionally a chain-of-thought reasoning trace.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred == gold:
            correct += 1
        elif is_reasoning_model:
            if llm_judge(pred, gold):
                correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Unstructured chain-of-thought outputs from reasoning models cannot be reliably parsed with regex; an LLM-as-a-Judge is required to extract the final answer.
• Evaluation scores vary significantly depending on the output protocol used (direct answer vs. regex vs. JSON vs. LLM judge), so consistency in parsing strategy is critical.
• Models often default to ego-centric viewpoints, leading to artificially low scores on perspective-taking tasks unless explicitly prompted for novel-view reasoning.

Evidence (verbatim from paper)

We measure accuracy on multiple-choice questions. For standard proprietary and open-source models, we test four output protocols: direct answer, regular-expression parsing, JSON parsing, and LLM-as-a-Judge*(Zheng et al., [2023])*. For reasoning-oriented models with unstructured CoT outputs, correctness is assessed by GPT-4.1-mini against ground truth.

Citation

@misc{jia2025omnispatial,
  title={OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models},
  author={Jia et al. (2025)},
  year={2025},
  note={arXiv:2506.03135}
}

• arXiv: 2506.03135