Spatial Dise Eval

spatial-dise-eval

Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models — Huang et al. (2025) (arXiv:2510.13394, 2025)

What this evaluates

Evaluates spatial reasoning capabilities in vision-language models across a 2x2 cognitive taxonomy (Intrinsic/Extrinsic × Static/Dynamic). It probes mental rotation, multi-step 3D transformations, and dynamic scene simulation using synthetically rendered 3D VQA pairs.

Datasets

• Spatial-DISE — total 12000; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Multiple-choice accuracy computed via exact match between the model's predicted option and the ground truth answer. Malformed or verbose model outputs are first parsed using an external LLM (Deepseek-R1) before comparison.

Input / output format

Input: 3D scene image (synthetically rendered via Blender) paired with a multiple-choice question text probing spatial relationships, transformations, or mental simulation.

Output: A single letter/option corresponding to the correct answer, optionally preceded by reasoning text.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        parsed_pred = parse_answer(pred)  # e.g., via Deepseek-R1
        if parsed_pred == gold:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Model outputs are frequently malformed or verbose, requiring an external parser (Deepseek-R1) before exact-match scoring can be reliably applied.
• Human baseline performance is averaged across a minimum of three unique participants per question, introducing inter-subject variance not captured by standard automated metrics.
• Fine-tuning on the Spatial-DISE dataset can cause catastrophic forgetting on other spatial benchmarks (e.g., CVBench, SAT), complicating claims about generalization.

Evidence (verbatim from paper)

We evaluate multiple-choice accuracy using exact match via the VLMEvalKit. Deepseek-R1 is used to parse answers from malformed model outputs.

Citation

@misc{huang2025spatialdise,
  title={Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2510.13394}
}

• arXiv: 2510.13394