Gsr Bench Eval

gsr-bench-eval

GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs — Rajabi et al. (2024) (arXiv:2406.13246, 2024)

What this evaluates

Evaluates multimodal LLMs' ability to understand and disambiguate spatial relations (e.g., on, under, left of, right of, in front of, behind) between objects in images. It isolates spatial reasoning from object grounding by providing depth maps, bounding boxes, and segmentation masks alongside images.

Datasets

• GSR-Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted spatial relations out of total test instances.

Input / output format

Input: RGB image, spatial relation query/clause (e.g., 'Is object A on object B?'), and optionally depth map, bounding box coordinates, and segmentation masks.

Output: Text prediction of the spatial relation (e.g., 'on', 'under', 'left of', 'right of', 'in front of', 'behind').

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return (correct / len(gold_labels)) * 100

Common pitfalls

• Confusing spatial relations when depth cues are absent, leading to front/back or left/right disambiguation errors.
• Models may rely on object grounding priors rather than true spatial reasoning if masks/boxes are not properly conditioned.
• Prompting strategies and depth-augmentation significantly impact scores, making cross-model comparisons sensitive to input formatting.

Evidence (verbatim from paper)

LLaMA-3-LLaVA-NeXT-8B achieves 86.1% accuracy, outperforming XVLM-COCO (60.4%) and demonstrating strong scalability with depth-augmented prompting that improves disambiguation of front/back and left/right relations.

Citation

@misc{rajabi2024gsrbench,
  title={GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs},
  author={Rajabi et al. (2024)},
  year={2024},
  note={arXiv:2406.13246}
}

• arXiv: 2406.13246