Vsi Bench Eval

vsi-bench-eval

Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations — Yuan et al. (2026) (arXiv:2603.08592, 2026)

What this evaluates

Evaluates multimodal large language models' ability to reason about 3D spatial relationships from visual inputs. It probes capabilities across configurational tasks (counting, relative distance/direction, route planning), measurement estimation (object/room size, absolute distance), and spatiotemporal ordering.

Datasets

• VSI-Bench — total ?; splits: test (-1)

Metrics

• score (primary) — range: percent
  • Percentage of correctly answered questions per subtask (object counting, distance, direction, size, route planning, order of appearance), averaged across all subtasks for the overall score.

Input / output format

Input: Annotated images with numeric object IDs, textual references encoding 3D geometric attributes (bounding boxes, shape primitives, room size), and a prompt template specifying coordinate system handedness and upright orientation.

Output: Natural language answers to spatial reasoning questions, often requiring step-by-step reasoning or direct numerical/categorical responses.

Scoring recipe

def compute_vsi_bench_score(predictions, gold):
    task_scores = {}
    for task in ['obj_count', 'abs_dist', 'obj_size', 'room_size', 'rel_dist', 'rel_dir', 'route_plan', 'appr_order']:
        correct = sum(1 for p, g in zip(predictions[task], gold[task]) if p == g)
        task_scores[task] = (correct / len(gold[task])) * 100
    avg_score = sum(task_scores.values()) / len(task_scores)
    return task_scores, avg_score

Common pitfalls

• Voxel-based object separation struggles in cluttered scenes, leading to inaccurate object counts and size estimations.
• Point clouds reconstructed from images are noisy, often causing overestimated bounding box dimensions.
• Benchmarks assume dense video frames, but real-world scenarios often provide only sparse views, requiring robust multi-view reasoning.

Evidence (verbatim from paper)

Among all MLLMs, GPT-5 achieves the highest overall score. While the fine-tuned VG LLM attains a comparable performance, it should be noted that its training data include videos overlapping with VSI-Bench and questions similar to those in the benchmark.

Citation

@misc{yuan2026boosting,
  title={Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations},
  author={Yuan et al. (2026)},
  year={2026},
  note={arXiv:2603.08592}
}

• arXiv: 2603.08592