qhjqhj00 multihopspatial-eval
MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model — Lee et al. (2026) (arXiv:2603.18892, 2026)
What this evaluates
Probes a vision-language model's ability to perform multi-hop compositional spatial reasoning and precise visual grounding. It tests whether models can correctly answer complex, multi-step spatial queries while simultaneously localizing the target object with high bounding box accuracy.
Datasets
- MultihopSpatial — total ?; splits: test (-1)
Metrics
MCQ Accuracy— range: percent- Percentage of instances where the predicted multiple-choice answer exactly matches the ground truth answer ($\hat{y} = y^*$).
Acc@50IoU(primary) — range: percent- Percentage of instances where the predicted answer matches the ground truth AND the Intersection over Union (IoU) between the predicted and ground truth bounding boxes is at least 0.5.
Avg. IoU— range: [0, 1]- Mean IoU between predicted and ground truth bounding boxes, computed exclusively over the subset of instances where the multiple-choice answer was correct.
Input / output format
Input: An image and a multiple-choice question requiring 1–3 hop compositional spatial reasoning (e.g., attribute, position, relation queries from ego- or exo-centric perspectives).
Output: A selected multiple-choice answer and a bounding box prediction for the target object.
Scoring recipe
mcq_correct = (pred_answer == gold_answer)
iou = calculate_iou(pred_bbox, gold_bbox)
grounded_correct = mcq_correct and (iou >= 0.5)
mcq_acc = mean(mcq_correct) * 100
acc_at_50iou = mean(grounded_correct) * 100
avg_iou = mean(iou[mcq_correct]) if sum(mcq_correct) > 0 else 0.0
Common pitfalls
- Relying solely on MCQ accuracy is highly misleading, as models often produce shortcut-based answers without genuine spatial localization.
- Ego-centric evaluation acts as a blind spot that compresses performance to a 20–25% floor, masking grounding disparities that are clearly visible under exo-centric conditions.
- Assuming test-time reasoning (e.g., chain-of-thought) scales linearly with complexity; performance degrades steeply at 3-hop tasks, showing diminishing returns.
Evidence (verbatim from paper)
We employ three complementary metrics to jointly evaluate reasoning correctness and spatial grounding: MCQ Accuracy. Measures the percentage of correct multiple-choice predictions $(\hat{y} = y^{})$. While standard, it does not verify spatial localization. Acc@50IoU. Our primary grounded metric requires correct answer selection and precise localization. A prediction is correct only if $\hat{y} = y^{}$ and IoU $(\hat{B}, B^{*}) \geq 0.5$. This filters out ungrounded predictions, ensuring genuine localization. Avg. IoU. Computed exclusively over MCQ-correct samples, this metric isolates grounding capability from reasoning errors, evaluating how precisely a model localizes the target once correctly identified.
Citation
@misc{lee2026multihopspatial,
title={MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model},
author={Lee et al. (2026)},
year={2026},
note={arXiv:2603.18892}
}
- arXiv: 2603.18892