Multimodal Math Reasoning Eval

multimodal-math-reasoning-eval

DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage — Gao et al. (2026) (arXiv:2603.01106, 2026)

What this evaluates

Evaluates the ability of multimodal large language models to solve mathematical problems that require interpreting visual diagrams alongside textual prompts. It probes complex reasoning capabilities across diverse difficulty levels and languages (English and Chinese).

Datasets

• MathVista — total ?; splits: test (-1)
• MathVerse — total ?; splits: test (-1)
• MathVision — total ?; splits: test (-1)
• OlympiadBench — total ?; splits: test (-1)
• WeMath — total ?; splits: test (-1)
• MMK12-test — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered instances, calculated as the number of exact matches between the model's predicted answer and the ground truth divided by the total number of test instances.

Input / output format

Input: Multimodal mathematical problems consisting of an image (diagram/figure) and a textual question.

Output: Textual reasoning steps followed by a final answer.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• The benchmarks are multimodal, so evaluating text-only models or ignoring image perturbations/variants leads to unfair comparisons.
• The paper uses a subset (5,000 instances) for ablation studies on MMK12, not the full test set, which may affect generalization claims.
• Accuracy is reported as a simple average across benchmarks, which might mask performance disparities on specific domains or languages.

Evidence (verbatim from paper)

As reported in Table 2, removing any single component consistently decreases performance, with the full DIVA-GRPO model achieving the highest accuracy across all benchmarks. These results indicate that all components contribute complementary gains and none can be omitted without performance degradation, highlighting the necessity of the complete model design.

Citation

@misc{gao2026divagrp,
  title={DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2603.01106}
}

• arXiv: 2603.01106