Mathreal Eval

mathreal-eval

MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models — Feng et al. (2025) (arXiv:2508.06009, 2025)

What this evaluates

Evaluates the ability of multimodal large language models to perform K-12 mathematical reasoning on real-world, mobile-captured images. It probes robustness to visual degradation (blur, rotation, handwritten annotations) and perspective variations, measuring how well models extract text and figures to solve math problems under imperfect conditions.

Datasets

• MathReal — total 2000; splits: testmini (480), test (1520); repo https://github.com/junfeng0288/MathReal

Metrics

• Strict Accuracy ($ ext{Acc}_{ ext{str}}$) — range: [0, 1]
  • 1 if all sub-answers within a question are correct, 0 otherwise.
• Loose Accuracy (Acc) (primary) — range: [0, 1]
  • Average proportion of correctly answered sub-questions across all questions.

Input / output format

Input: Single image containing a K-12 math problem with text and figures. Evaluations may use image-only (I), image + ground-truth question text (I+QG), or image + ground-truth text + figure description (I+QG+DG).

Output: Chain-of-thought reasoning followed by the final answer (0-shot).

Scoring recipe

def compute_accuracy(predictions, golds):
    correct_count = 0
    for pred, gold in zip(predictions, golds):
        # Automated pipeline uses GPT-4.1-nano to compare pred vs gold
        # Enforces: mathematical equivalence, numerical tolerance, unit consistency, symbolic structure
        is_match = gpt4_nano_evaluate(pred, gold)
        correct_count += is_match
    return correct_count / len(predictions)

Common pitfalls

• Strict accuracy penalizes any incorrect sub-answer, so minor OCR or formatting errors cause full question failure.
• Real-world visual degradation (blur, rotation, handwritten annotations) severely impairs visual perception, leading to performance drops far below clean-image benchmarks.

Evidence (verbatim from paper)

Strict Accuracy ($ ext{Acc}_{ ext{str}}$) requires that all sub-answers within a question be correct for the model to receive credit. If any sub-answer is incorrect, the entire question is marked wrong. Loose Accuracy (Acc) allows partial correctness and is computed as the proportion of correctly answered sub-questions within each question. For both metrics, an automated scoring pipeline based on GPT-4.1-nano compares model answers against reference answers, enforcing strict rules for mathematical equivalence, numerical tolerance, unit consistency, and symbolic structure to ensure scalable and reliable evaluation in real-world tasks.

Citation

@misc{feng2025mathreal,
  title={MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models},
  author={Feng et al. (2025)},
  year={2025},
  note={arXiv:2508.06009}
}

• arXiv: 2508.06009