Mmr1 Eval

mmr1-eval

MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources — Sicong Leng et al. (arXiv:2509.21268, 2025)

What this evaluates

Evaluates multimodal mathematical and logical reasoning capabilities of vision-language models. It probes complex multi-step problem solving, visual reasoning, logical deduction, and chart-based understanding across five diverse benchmarks.

Datasets

• MathVerse — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• MathVision — total ?; splits: test (-1)
• LogicVista — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions per benchmark, averaged across MathVerse, MathVista, MathVision, LogicVista, and ChartQA.

Input / output format

Input: Multimodal prompts containing an image and a mathematical or logical question.

Output: A step-by-step reasoning chain followed by the final answer enclosed in \box{}.

Scoring recipe

def score(predictions, golds, benchmark):
    if benchmark in ['MathVision', 'MMMU']:
        return official_protocol_score(predictions, golds)
    if is_math_question(benchmark):
        return math_verify_score(predictions, golds)
    else:
        return exact_match_or_semantic_score(predictions, golds)

# Final metric is the average accuracy across the 5 benchmarks.

Common pitfalls

• Benchmarks have varying official protocols (e.g., MathVision, MMMU) that must be strictly followed rather than applying a single uniform parser.
• Math questions require specialized verifiers (Math-Verify/MathRuler) while non-math questions rely on exact matching, requiring careful routing.
• The \box{} delimiter must be correctly extracted; missing it causes false negatives even if the answer is present in the text.

Evidence (verbatim from paper)

We adopt a unified prompt across all evaluations, requiring models to enclose final answers in “\box{}” (full prompt in Appendix[D]). Inference is performed using vLLM for efficient generation. For benchmarks with official protocols (e.g., MathVision, MMMU), we strictly follow the original procedures. For others, mathematical questions are assessed with Math-Verify (Kydlíček, [2025]) and MathRuler (hiyouga, [2025]), while non-mathematical ones use exact matching. To ensure robustness, we further (1) select the most semantically similar option when multiple-choice answers do not exactly match any candidate, and (2) employ GPT-4o (OpenAI, [2024]) as an auxiliary judge for open-ended questions where exact matching or extraction fails.

Citation

@misc{leng2025mmr1,
  title={MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources},
  author={Sicong Leng et al.},
  year={2025},
  note={arXiv:2509.21268}
}

• arXiv: 2509.21268