qhjqhj00 spatialqa-eval
SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models — Xie et al. (2026) (arXiv:2602.20901, 2026)
What this evaluates
Evaluates the ability of vision-language models to perform multi-step spatial logical reasoning by tracking object dependencies and understanding scene layouts across real-world indoor environments.
Datasets
- SpatiaLQA — total 9605; splits: test (-1); repo https://github.com/xieyc99/SpatiaLQA
Metrics
accuracy(primary) — range: [0, 1]- Standard exact-match accuracy for closed-form QA: 1 if the model's predicted answer matches the ground truth answer (case-insensitive, normalized whitespace), 0 otherwise. Averaged over all 9,605 samples.
Input / output format
Input: A single RGB image of an indoor scene and a natural language question describing a spatial relationship or logical query about objects within that scene.
Output: A natural language text answer corresponding to the question.
Scoring recipe
def compute_accuracy(predictions, gold_answers):
correct = 0
for pred, gold in zip(predictions, gold_answers):
if normalize(pred) == normalize(gold):
correct += 1
return correct / len(gold_answers)
Common pitfalls
- Counting different camera angles or lighting changes of the same physical setup as distinct scenes, which violates the dataset's scene definition rules.
- Allowing the same object to appear more than ten times within a single scene, which breaks the designed diversity and frequency constraints.
- Failing to verify step validity and prerequisite correctness during the annotation review cycle, leading to logically unsound QA pairs.
Evidence (verbatim from paper)
The benchmark comprises 9,605 QA pairs from 241 real-world indoor scenes, constructed via manual annotation and graph-augmentation techniques. Each sample annotation consists of four components: the question, the answer, the corresponding image path, and the associated scene category.
Citation
@misc{xie2026spatialqa,
title={SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models},
author={Xie et al. (2026)},
year={2026},
note={arXiv:2602.20901}
}
- arXiv: 2602.20901