Multimodal Visual Reasoning Eval

multimodal-visual-reasoning-eval

Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning — Hai-Long Sun et al. (arXiv:2503.13360, 2025)

What this evaluates

This evaluation probes a model's ability to perform long-chain, multi-modal reasoning on mathematical problems that require deep visual understanding. It measures how well the model integrates image evidence with textual reasoning steps to arrive at correct answers across diverse math benchmarks.

Datasets

• MathVista — total 6141; splits: testmini (-1)
• MathVision — total 3040; splits: test (-1)
• MathVerse — total 2612; splits: testmini (-1)
• Dynamath — total ?; splits: test (-1)
• OlympiadBench — total 8476; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered problems, evaluated automatically using GPT-4o-mini against ground truth answers.

Input / output format

Input: A single image paired with a mathematical problem statement requiring multi-step reasoning.

Output: A chain-of-thought reasoning trace followed by a final answer.

Scoring recipe

predictions = model.generate(image, prompt)
scores = []
for pred, gold in zip(predictions, gold_answers):
    judge_prompt = f'Question: {prompt}\nModel Answer: {pred}\nGround Truth: {gold}\nIs the model answer correct? Answer Yes or No.'
    response = gpt4o_mini(judge_prompt)
    scores.append(1 if 'Yes' in response else 0)
accuracy = sum(scores) / len(scores) * 100

Common pitfalls

• Failing to exclude the text-only split from MathVerse and theorem-proof sections from OlympiadBench, which inflates scores by removing visual requirements.
• Evaluating on the full test set instead of the specified testmini splits for MathVerse and MathVista, leading to unfair comparisons.
• Assuming human grading is used; the paper explicitly relies on GPT-4o-mini for automated evaluation.

Evidence (verbatim from paper)

These benchmarks evaluate problem-solving abilities in mathematics, and following standard practice, we use GPT-4o-mini as the evaluator. Following the VLMEvalKit guidelines, we exclude the text-only split from MathVerse and the theorem-proof sections from OlympiadBench. For a fair comparison, we conduct evaluations using the testmini sets of MathVerse and MathVista.

Citation

@misc{sun2025mitigatingvisualforgetting,
  title={Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning},
  author={Hai-Long Sun et al.},
  year={2025},
  note={arXiv:2503.13360}
}

• arXiv: 2503.13360