Revqa Spatial Reasoning Eval

revqa-spatial-reasoning-eval

REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models — Chatterjee et al. (2024) (arXiv:2408.02231, 2024)

What this evaluates

Evaluates the spatial reasoning and logical comprehension capabilities of multimodal large language models (MLLMs) on synthetic, spatially precise images. It probes robustness to negations, logical operators (AND/OR), adversarial object substitutions, and complex spatial relationships.

Datasets

• RevQA — total 50000; splits: test (50000)

Metrics

• performance (primary) — range: percent
  • Percentage of correct binary (Yes/No) predictions across all question types. Calculated as (number of correct answers / total number of questions) * 100.

Input / output format

Input: A synthetic 3D-rendered image paired with a natural language question (yes/no format) testing spatial relationships and logical operators.

Output: Binary response: 'Yes' or 'No'.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() in ['yes', 'no'] and pred.strip().lower() == gold.strip().lower():
        correct += 1
    total += 1
accuracy = (correct / total) * 100 if total > 0 else 0

Common pitfalls

• Models may exhibit order bias when answering combined questions; the benchmark mitigates this by randomizing clause order, but evaluators should ensure consistent ordering or account for it.
• Evaluation requires deterministic decoding (temperature=0) to avoid stochasticity masking true reasoning capabilities, especially for negation and adversarial questions.
• Binary responses must be strictly parsed as 'Yes' or 'No'; free-form text generation will break the performance calculation.

Evidence (verbatim from paper)

We instruct all models to generate binary responses and set the temperature =0, to remove stochasticity in the generated responses. We present our evaluation results in Table 7 and find that all models have a large gap in performance in reasoning over spatial relationships.

Citation

@misc{chatterjee2024revision,
  title={REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models},
  author={Chatterjee et al. (2024)},
  year={2024},
  note={arXiv:2408.02231}
}

• arXiv: 2408.02231