Spatial Reasoning Eval

spatial_reasoning_eval

Enhancing Spatial Reasoning in Vision-Language Models via Chain-of-Thought Prompting and Reinforcement Learning — Binbin Ji et al. (arXiv:2507.13362, 2025)

What this evaluates

Evaluates vision-language models' ability to count objects and reason about spatial relationships (depth, distance, relative position) in images. It probes segmentation capabilities, attention alignment, and robustness to linguistic variations (out-of-distribution shifts).

Datasets

• CLEVR_CoGenT_ValB — total ?; splits: val (-1)
• CVBench — total ?; splits: test (-1)
• Pixmo-Count — total ?; splits: test (-1)
• Static Spatial Reasoning (SAT) — total ?; splits: val (-1)
• VSR — total ?; splits: val (-1)
• VC Bench — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • The percentage of predictions that exactly match the ground truth count or answer.
• Close-Call Percentage — range: percent
  • The percentage of incorrect predictions that are off by exactly one: |{i:|ŷ_i−y_i|=1 and ŷ_i≠y_i}| / |{i:ŷ_i≠y_i}| × 100%.
• Pass@1 accuracy — range: percent
  • Accuracy computed over a single generation per query.
• Pass@4 accuracy — range: percent
  • Accuracy computed as the average correctness over four independent generations per query.

Input / output format

Input: RGB image and a natural language question (e.g., 'Which object is closer to the camera?' or 'How many red cubes are there?').

Output: A single predicted answer (e.g., object ID/name or count), or a structured Scene Graph followed by the answer when using Chain-of-Thought prompting.

Scoring recipe

def compute_metrics(predictions, ground_truths):
    correct = sum(1 for p, g in zip(predictions, ground_truths) if p == g)
    off_by_one = sum(1 for p, g in zip(predictions, ground_truths) if p != g and abs(p - g) == 1)
    incorrect = sum(1 for p, g in zip(predictions, ground_truths) if p != g)
    accuracy = (correct / len(predictions)) * 100
    close_call_pct = (off_by_one / incorrect) * 100 if incorrect > 0 else 0.0
    return accuracy, close_call_pct

Common pitfalls

• Confusing Pass@1 (single generation accuracy) with Pass@4 (average accuracy over 4 samples), which show different trade-offs between SFT and GRPO.
• Overlooking the out-of-distribution (OOD) evaluation protocol, which requires rephrasing test queries from 'close to' to 'far from' to test semantic generalization rather than surface memorization.
• Assuming standard Chain-of-Thought improves performance; the paper finds naive CoT degrades spatial reasoning and requires a structured two-step Scene Graph prompt to avoid reward hacking.

Evidence (verbatim from paper)

To test this, we evaluate performance using two complementary metrics on our counting dataset: Accuracy: The percentage of predictions that exactly match the ground truth count. Close-Call Percentage: Defined as |{i:|ŷ_i−y_i|=1 and ŷ_i≠y_i}| / |{i:ŷ_i≠y_i}| × 100%, where ŷ_i is the model’s predicted count and y_i is the ground truth.

Citation

@misc{ji2025enhancing,
  title={Enhancing Spatial Reasoning in Vision-Language Models via Chain-of-Thought Prompting and Reinforcement Learning},
  author={Binbin Ji et al.},
  year={2025},
  note={arXiv:2507.13362}
}

• arXiv: 2507.13362