Spatialthinker Eval

spatialthinker-eval

SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards — Hunar Batra et al. (arXiv:2511.07403, 2025)

What this evaluates

Evaluates multimodal LLMs on 3D spatial reasoning, depth/distance estimation, and general visual question answering. It probes the model's ability to ground objects in 3D space, understand spatial relations, and generalize to real-world VQA tasks using only RGB inputs.

Datasets

• SpatialThinker Evaluation Suite (12 VQA Benchmarks) — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Computed as the number of exact matches between the model's generated answer and the ground-truth answer divided by the total number of instances.

Input / output format

Input: RGB image paired with a natural language question.

Output: Natural language text answer.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Evaluated strictly zero-shot with greedy decoding (temperature=0.0), not few-shot or with sampling.
• SpatialRGPT uses depth inputs while all other baselines use only RGB, creating an input modality mismatch.
• Proprietary models (GPT-4o, Claude) use their default system prompts, which may differ from open-source model prompting.

Evidence (verbatim from paper)

All models are evaluated in a zero-shot setting using greedy decoding (temperature = 0.0). Accuracy is the primary evaluation metric.

Citation

@misc{batra2025spatialthinker,
  title={SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards},
  author={Hunar Batra et al.},
  year={2025},
  note={arXiv:2511.07403}
}

• arXiv: 2511.07403