Visual Spatial Reasoning Eval

visual-spatial-reasoning-eval

Visual Spatial Reasoning — Liu et al. (2022) (arXiv:2205.00363, 2022)

What this evaluates

This benchmark evaluates visual language models' ability to understand and reason about spatial relationships between objects in images. It specifically probes orientation-dependent relations, frame-of-reference shifts (intrinsic vs. relative), and zero-shot generalization to unseen object concepts.

Datasets

• VSR — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/cambridgeltl/visual-spatial-reasoning

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted True/False labels out of the total number of test instances. Calculated as (number of correct predictions) / (total predictions).

Input / output format

Input: Image-text pairs, where the text is a caption describing a spatial relation between objects in the image.

Output: Binary classification: True or False, indicating whether the caption accurately describes the spatial relation in the image.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
return correct / total

Common pitfalls

• Model performance is sensitive to random seeds; results should be averaged over at least three runs with different seeds.
• The zero-shot split is significantly harder than the random split due to unseen object concepts, not just fewer training examples.
• Proximity relations (e.g., 'near', 'far') are highly concept-dependent and may plateau or drop to chance level in zero-shot settings.

Evidence (verbatim from paper)

As shown in Table 4, the best-performing models on the random split are LXMERT and ViLT, reaching around 70% accuracy while VisualBERT is just slightly better than the chance level. On the zero-shot split, all models’ performance decline substantially and the best model ViLT only obtains 63.0% accuracy.

Citation

@misc{liu2022visualspatialreasoning,
  title={Visual Spatial Reasoning},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2205.00363}
}

• arXiv: 2205.00363