Sparrta Eval

sparrta-eval

SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models — Kargin et al. (2026) (arXiv:2601.11729, 2026)

What this evaluates

This benchmark evaluates the spatial reasoning capabilities of Visual Foundation Models (VFMs) by testing their ability to recognize spatial relations between object triples in synthetic images. It specifically probes both egocentric (camera-perspective) and allocentric (world-perspective) spatial understanding across diverse semantic objects and environments.

Datasets

• SpaRRTa — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted spatial relations (source-target-viewpoint triples) on the test set. Probes are trained per object triple and environment, and results are averaged across 2 random seeds and 3 distinct object triples per environment.

Input / output format

Input: A single RGB image (224×224) depicting a layout of three objects (source, target, viewpoint), processed by a frozen VFM backbone to extract patch tokens or global features.

Output: A categorical label indicating the spatial relation between the specified source, target, and viewpoint objects.

Scoring recipe

correct = 0
total = 0
for image, true_relation in test_set:
    features = frozen_vfm(image)
    probe_output = probe_head(features)
    predicted_relation = argmax(probe_output)
    if predicted_relation == true_relation:
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Using global average pooling (Linear probing) discards crucial patch-level spatial information, leading to artificially low performance compared to attention-based or multi-query pooling methods.
• Assuming 3D-supervised models (like VGGT) will universally outperform 2D self-supervised models; in reality, 3D supervision primarily enriches patch-level structure rather than global representations, making advanced probing necessary to observe gains.
• Confusing egocentric and allocentric tasks; allocentric recognition is consistently more challenging and yields systematically lower accuracy across all models and probing strategies.

Evidence (verbatim from paper)

For a given VFM, we evaluate the SpaRRTa performance in terms of the accuracy of probes trained for recognizing the spatial relations of given source, target, viewpoint object triples. Performance on SpaRRTa across images belonging to a given semantic triple therefore reflects the ability to resolve spatial relations for those particular object types, while averaging over diverse semantic triples provides an aggregate measure of spatial relation recognition. For each such triple of objects, we curate a dataset of images depicting different object layouts, and split it into train, validation, and test folds in 80/10/10 proportions. We train triple-specific probes for the number of epochs specified in Table 2, using the validation set to select the best-performing probe parameters. We report the test accuracies of probes, and repeat this procedure with 2 random seeds and a diverse selection of 3 distinct object triples per each of the 5 environments (see Section 3.3) to achieve a robust evaluation.

Citation

@misc{kargin2026sparrta,
  title={SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models},
  author={Kargin et al. (2026)},
  year={2026},
  note={arXiv:2601.11729}
}

• arXiv: 2601.11729