qhjqhj00 spatial-intelligence-eval
Scaling Spatial Intelligence with Multimodal Foundation Models — Cai et al. (2025) (arXiv:2511.13719, 2025)
What this evaluates
Evaluates a model's ability to perceive, reason about, and reconstruct 3D spatial layouts, multi-view relationships, and perspective-taking from visual inputs. It probes robustness against language shortcuts and tests generalization to longer video sequences and embodied manipulation tasks.
Datasets
- VSI-Bench — total ?; splits: test (-1)
- MMSI-Bench — total ?; splits: test (-1)
- MindCube — total ?; splits: train (10000), test (-1)
- ViewSpatial-Bench — total ?; splits: test (-1)
- SITE — total ?; splits: test (-1)
- MMBench-En — total ?; splits: test (-1)
- EmbodiedBench (spatial subset) — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Standard multiple-choice accuracy: the percentage of questions where the model's predicted option exactly matches the ground-truth label.
success rate— range: percent- The proportion of robot manipulation tasks completed successfully within the allowed steps or constraints.
Input / output format
Input: Visual inputs (images or uniformly sampled video frames, e.g., 32 frames for VSI-Bench) paired with multiple-choice questions or spatial instructions. For robot tasks, images with bounding-box coordinates or enriched spatial prompts.
Output: Multiple-choice letter (A/B/C/D) or binary success/failure for robot manipulation tasks.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p.strip().upper() == g.strip().upper())
return correct / len(gold) * 100
def compute_success_rate(success_flags):
return sum(success_flags) / len(success_flags) * 100
Common pitfalls
- Models may exploit language shortcuts or text priors instead of visual reasoning (mitigated by VSI-Debiased and no-vision tests).
- Performance can be artificially inflated by overfitting to answer option ordering (addressed via soft/hard circular tests).
- Training on short video sequences (≤16 frames) may not guarantee robust extrapolation to longer contexts at inference.
Evidence (verbatim from paper)
We report success rates under two prompting settings: the official prompt (OP) and a spatial-intelligence-oriented prompt (SIP). OP supplies bounding-box coordinates extracted from the input image, whereas SIP enriches OP with additional object-grounding cues to reduce ambiguity from object recognition and better isolate spatial-reasoning performance. Across both OP and SIP, SenseNova-SI delivers substantial improvements, demonstrating that enhanced spatial intelligence directly benefits embodied manipulation.
Citation
@misc{cai2025scalingspatial,
title={Scaling Spatial Intelligence with Multimodal Foundation Models},
author={Cai et al. (2025)},
year={2025},
note={arXiv:2511.13719}
}
- arXiv: 2511.13719