Vlm Interaction Reasoning Eval

vlm-interaction-reasoning-eval

Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning — Dayong Liang et al. (2025) (arXiv:2505.09118, 2025)

What this evaluates

Evaluates vision-language models on general visual understanding, spatial/relational reasoning, and specifically interactional reasoning in dynamic scenes using a suite of standard VQA and scene understanding benchmarks.

Datasets

• VQAv2 — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• VSR — total ?; splits: test (-1)
• RealWorldQA — total ?; splits: test (-1)
• MMT-Bench — total ?; splits: test (-1)
• SEEDBench — total ?; splits: test (-1)
• A-Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Exact-match accuracy: the percentage of predictions that exactly match the ground truth answer after lowercasing and stripping whitespace.

Input / output format

Input: An image and a natural language question.

Output: A short text answer or phrase.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, golds):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return (correct / len(golds)) * 100

Common pitfalls

• Data leakage is likely, as the training data includes LLaVA-v1.5-mixed-665k and OpenImages, which overlap with test benchmarks like VQAv2 and GQA.
• Exact-match scoring on VQA benchmarks is highly sensitive to minor phrasing differences and ignores semantic equivalence.
• The paper reports average performance across benchmarks but does not provide statistical significance testing or confidence intervals.

Evidence (verbatim from paper)

We carefully selected a diverse suite of benchmarks to comprehensively evaluate both general vision-language capabilities and specific interactional reasoning skills: General VL Understanding: VQAv2, VizWiz, and TextVQA... Spatial & Relational Understanding: GQA, VSR... Real-world Interaction Understanding: RealWorldQA, MMT-Bench... Compositional Reasoning: SEEDBench, A-Bench

Citation

@misc{liang2025seeing,
  title={Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning},
  author={Dayong Liang et al. (2025)},
  year={2025},
  note={arXiv:2505.09118}
}

• arXiv: 2505.09118