# Visual Spatial Reasoning Eval

> This benchmark evaluates visual language models' ability to understand and reason about spatial relationships between objects in images. It specifically probes orientation-dependent relations, frame-of-reference shifts (intrinsic vs. relative), and zero-shot generalization to unseen object concepts. Use when the user wants to benchmark on VSR, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/visual-spatial-reasoning-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/visual-spatial-reasoning-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/visual-spatial-reasoning-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/visual-spatial-reasoning-eval

---


# visual-spatial-reasoning-eval

> Visual Spatial Reasoning — Liu et al. (2022) (arXiv:2205.00363, 2022)

## What this evaluates

This benchmark evaluates visual language models' ability to understand and reason about spatial relationships between objects in images. It specifically probes orientation-dependent relations, frame-of-reference shifts (intrinsic vs. relative), and zero-shot generalization to unseen object concepts.

## Datasets

- **VSR** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/cambridgeltl/visual-spatial-reasoning

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted True/False labels out of the total number of test instances. Calculated as (number of correct predictions) / (total predictions).

## Input / output format

**Input**: Image-text pairs, where the text is a caption describing a spatial relation between objects in the image.

**Output**: Binary classification: True or False, indicating whether the caption accurately describes the spatial relation in the image.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Model performance is sensitive to random seeds; results should be averaged over at least three runs with different seeds.
- The zero-shot split is significantly harder than the random split due to unseen object concepts, not just fewer training examples.
- Proximity relations (e.g., 'near', 'far') are highly concept-dependent and may plateau or drop to chance level in zero-shot settings.

## Evidence (verbatim from paper)

> As shown in Table 4, the best-performing models on the random split are LXMERT and ViLT, reaching around 70% accuracy while VisualBERT is just slightly better than the chance level. On the zero-shot split, all models’ performance decline substantially and the best model ViLT only obtains 63.0% accuracy.

## Citation

```bibtex
@misc{liu2022visualspatialreasoning,
  title={Visual Spatial Reasoning},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2205.00363}
}
```

- arXiv: 2205.00363