qhjqhj00 math-vision-eval
Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset — Wang et al. (2024) (arXiv:2402.14804, 2024)
What this evaluates
Evaluates multimodal mathematical reasoning capabilities of LLMs and LMMs on problems with visual contexts. Probes geometric invariance, spatial reasoning, and deep mathematical reasoning across 16 disciplines and 5 difficulty levels.
Datasets
- MATH-V — total 3040; splits: testmini (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions out of the total number of questions in the dataset.
Input / output format
Input: A mathematical problem presented with an accompanying image (visual context), provided in a zero-shot or Chain-of-Thought prompt format.
Output: A final answer to the mathematical problem, either as a multiple-choice selection or an open-ended solution, depending on the question type.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
return (correct / len(gold)) * 100
Common pitfalls
- Chain-of-Thought prompting does not consistently improve performance and can actually decrease accuracy for some models (e.g., Gemini Pro, Qwen-VL-Max).
- Performance varies significantly across mathematical subjects (e.g., struggles in counting vs. arithmetic), so reporting only overall accuracy masks domain-specific weaknesses.
- Human baseline was evaluated on the 'testmini' subset only, which may not represent the full dataset difficulty.
Evidence (verbatim from paper)
Notably, the overall accuracy for the leading model, GPT-4V, is only 22.76%, which, while being the highest, still indicates a significant challenge in tackling the dataset.
Citation
@misc{wang2024mathvision,
title={Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset},
author={Wang et al. (2024)},
year={2024},
note={arXiv:2402.14804}
}
- arXiv: 2402.14804