qhjqhj00 mathnet-solve-eval
MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval — Shaden Alshammari et al. (arXiv:2604.18584, 2026)
What this evaluates
Evaluates a model's ability to solve Olympiad-level mathematical problems across multiple domains (algebra, geometry, combinatorics, number theory) and modalities (text and images). It measures whether models can produce consistent, correct reasoning rather than just guessing the final answer.
Datasets
- MathNet-Solve — total 6400; splits: test (6400); repo https://github.com/ShadeAlsha/MathNet
Metrics
Problem Solving Accuracy(primary) — range: percent- Binarized score from a GPT-5 judge. The judge scores the model's solution against the reference solution on a 0–7 scale. Scores ≥ 6 are marked correct (1), others incorrect (0). Accuracy is the percentage of correct solutions.
Input / output format
Input: Problem statement (text and/or image). For multimodal models, both text and image are provided; otherwise, a text-only description of the image is supplied.
Output: A generated solution to the mathematical problem.
Scoring recipe
score = gpt5_judge(problem_statement, reference_solution, model_output) # 0-7
is_correct = 1 if score >= 6 else 0
accuracy = sum(is_correct) / total_problems
Common pitfalls
- Relies on an LLM judge (GPT-5) for grading rather than exact string matching, which introduces potential judge bias or inconsistency.
- Binarizes a 0-7 scale at ≥ 6, so minor errors are penalized heavily compared to a lenient threshold.
- Evaluates both text-only and multimodal inputs; models must handle image descriptions if they lack native vision capabilities.
Evidence (verbatim from paper)
On MathNet-Solve, we evaluate two types of models: (i) LLMs and LMMs... For each problem, the judge model is provided with the problem statement, the reference solution, and the model-generated solution, and is asked to assess whether the output is consistent with the correct answer using a numeric score from 0–7. We binarize the score by marking outputs with score ≥ 6 as correct (fully correct or containing only minor errors) and scores <6 as incorrect. This allows us to distinguish between models that arrive at the correct final answer by coincidence versus those that demonstrate consistent reasoning ability. Table 3: Problem Solving Accuracy (%, ↑) on MathNet-Solve-Test (6,400 problems).
Citation
@misc{alshammari2026mathnet,
title={MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval},
author={Shaden Alshammari et al.},
year={2026},
note={arXiv:2604.18584}
}
- arXiv: 2604.18584