qhjqhj00 spce-10-eval
SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence — Gong et al. (2025) (arXiv:2506.07966, 2025)
What this evaluates
This benchmark evaluates multimodal large language models on compositional spatial intelligence by testing their ability to reason across 10 atomic spatial capabilities (e.g., counting, localization, spatial relations) combined into 8 complex tasks. It probes scene understanding using 2D and 3D inputs through multiple-choice questions, revealing how models handle hierarchical spatial reasoning and capability integration.
Datasets
- SpaCE-10 — total 5000; splits: test (5000); repo https://github.com/VisionXLab/SpaCE-10
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total questions) * 100. Applied uniformly to both single-answer and multiple-choice formats.
Input / output format
Input: Visual inputs (2D images or 3D point clouds) paired with multiple-choice questions targeting specific spatial reasoning capabilities.
Output: A selected answer option from the provided choices (single-answer or multiple-answer format).
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Random baseline is 25% for 4-choice questions, but the paper notes performance might be lower than random due to removing random assignment in LMMs-Eval.
- Smaller models tend to overfit to single-answer formats, drastically underperforming on multiple-answer tasks (<5% vs >30%).
- High overall accuracy does not necessarily reflect strong atomic spatial capabilities; capability breakdown is needed to avoid misleading conclusions.
Evidence (verbatim from paper)
For the 3 questions with integrating C7 (Multi-view Fusion) capability, the models’ accuracy in SA, OO, and OS tasks decreases by 19.1%, 7.6%, and 21.0%, respectively.
Citation
@misc{gong2025spce10,
title={SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence},
author={Gong et al. (2025)},
year={2025},
note={arXiv:2506.07966}
}
- arXiv: 2506.07966