Mmsi Bench Eval

mmsi-bench-eval

MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence — Yang et al. (2025) (arXiv:2505.23764, 2025)

What this evaluates

This benchmark evaluates multimodal large language models' ability to perform multi-image spatial reasoning. It probes capabilities such as tracking object and camera motion, reconstructing scenes from multiple views, and inferring spatial logic across image sequences.

Datasets

• MMSI-Bench — total 1000; splits: test (1000)

Metrics

• accuracy (primary) — range: percent
  • Exact match between answers extracted from model outputs and ground-truth answers for multiple-choice questions. An LLM-based fallback strategy extracts the intended response if the model fails to generate the answer in the required format.

Input / output format

Input: A set of multiple images accompanied by a multiple-choice question testing spatial relationships, motion tracking, or scene reconstruction.

Output: A single character/option corresponding to the correct choice from the provided multiple-choice options.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        extracted = extract_answer(pred)  # Uses LLM fallback if format fails
        if extracted == gold:
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Blind GPT-4o scores near random guessing, proving tasks require genuine visual-spatial reasoning rather than language priors or commonsense knowledge.
• Chain-of-Thought prompting often degrades performance, as models lack foundational spatial understanding to benefit from step-by-step reasoning.
• Increasing model parameters yields only marginal accuracy gains, indicating data quality/diversity is a bigger bottleneck than scale.

Evidence (verbatim from paper)

We report accuracy (%) using exact match between answers extracted from model outputs and ground-truth answers for our multiple-choice questions. If a model fails to generate an answer in the required format, we adopt the LLM-based fallback strategy from VLMEvalKit*(Duan et al., [2024])* to extract the intended response.

Citation

@misc{yang2025mmsibench,
  title={MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2505.23764}
}

• arXiv: 2505.23764