Sphere Eval

sphere-eval

SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation — Zhang et al. (2024) (arXiv:2412.12693, 2024)

What this evaluates

Evaluates spatial reasoning and visual understanding in vision-language models across a hierarchy of tasks, including single-skill perception (position, counting, distance, size), multi-skill integration, and complex physical-world reasoning (object occlusion and manipulation). It specifically probes egocentric vs. allocentric perspective taking and susceptibility to object hallucination.

Datasets

• SPHERE — total 2285; splits: test (2285); repo https://github.com/zwenyu/SPHERE-VLM

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered instances. For multiple-choice questions, it is the proportion of correct selections. For open-ended counting tasks, it is exact-match accuracy on the integer count.

Input / output format

Input: An image (sourced from MS COCO-2017 test split) paired with a natural language question. Questions are either multiple-choice (2-3 options) or open-ended numerical prompts.

Output: A natural language response: either the selected option text for MCQs or an integer for counting tasks.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if str(pred).strip().lower() == str(gold).strip().lower():
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Confusing egocentric (entity-relative) and allocentric (camera-relative) viewpoints when evaluating position tasks.
• Overlooking 'trick' counting questions where the ground truth is zero, which specifically tests model hallucination rather than mere counting ability.
• Failing to separate intermediate (perception-based) and final (reasoning-based) questions in occlusion/manipulation tasks, as they measure different cognitive stages.

Evidence (verbatim from paper)

We annotate a total of 2,285 question-answer pairs. ... Counting-related tasks require open-ended numerical responses, while other tasks use multiple-choice questions (MCQs) with two options by default and three options when additional clarity is needed. ... Table 3: Average accuracy (%) of VLMs on SPHERE tasks.

Citation

@misc{zhang2024sphere,
  title={SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2412.12693}
}

• arXiv: 2412.12693