qhjqhj00 vcbench-eval
Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency — Wang et al. (2025) (arXiv:2504.18589, 2025)
What this evaluates
Evaluates multimodal mathematical reasoning capabilities of vision-language models, specifically focusing on vision-centric elementary math problems that require explicit visual dependencies across multiple images. It probes spatial, temporal, geometric, logical, and pattern recognition skills to measure how well models integrate cross-modal information for compositional reasoning.
Datasets
- VCBench — total 1720; splits: test (1720)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered multiple-choice questions. Calculated as the number of correct predictions divided by the total number of questions, averaged across all 17 subtasks.
Input / output format
Input: A set of multiple images (average 3.9 per question) accompanied by a textual question. Questions are presented in a multiple-choice format.
Output: A single selected option from the provided multiple-choice answers.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Models often show significantly higher accuracy in single-image settings compared to multi-image settings, indicating a bias toward single-image optimization rather than true compositional reasoning.
- Math-specialist models (e.g., G-LLaVA, MathLlava) underperform general-purpose VLMs, contrary to expectations for math benchmarks.
- Performance varies drastically across difficulty levels (Easy/Medium/Hard), so reporting only average accuracy masks model weaknesses on complex reasoning tasks.
Evidence (verbatim from paper)
Table 5: Accuracy comparison of various models on questions categorized by difficulty along with their average performance.
Citation
@misc{wang2025vcbench,
title={Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency},
author={Wang et al. (2025)},
year={2025},
note={arXiv:2504.18589}
}
- arXiv: 2504.18589