Mathnet RAG Eval

mathnet-rag-eval

MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval — Shaden Alshammari et al. (arXiv:2604.18584, 2026)

What this evaluates

Evaluates how retrieval quality impacts downstream mathematical problem solving. It compares zero-shot performance against retrieval-augmented settings using either embedding-retrieved or expert-paired problems with their solutions.

Datasets

• MathNet-RAG — total 35; splits: test (35); repo https://github.com/ShadeAlsha/MathNet

Metrics

• Retrieval-Augmented Problem Solving Accuracy (primary) — range: percent
  • Percentage of correctly solved problems under three settings: Zero Shot, Embed-RAG (retrieves 1 problem via gemini-embedding-001), and Expert-RAG (provides expert-paired problem). Graded by humans or averaged LLM judge.

Input / output format

Input: Target problem statement. In RAG settings, the retrieved problem and its official solution are appended as context.

Output: A generated solution to the target problem.

Scoring recipe

context = get_context(setting) # retrieved problem + solution or None
model_output = generate(target_problem, context)
is_correct = human_or_llm_judge(target_problem, reference_solution, model_output)
accuracy = sum(is_correct) / total_problems

Common pitfalls

• Small test set (35 problems) leads to high standard errors in reported accuracy.
• Grading is done by humans or averaged LLM judges, introducing potential subjectivity or judge-model bias.
• Embed-RAG can sometimes hurt performance if the retriever returns near-miss problems that add noise.

Evidence (verbatim from paper)

To assess the impact of retrieval on downstream problem solving, we evaluate three settings. In Zero Shot, the model receives only the target problem. In Embed-RAG, we retrieve one related problem using gemini-embedding-001, then provide the retrieved problem and its official solution as additional context. In Expert-RAG, we replace the retrieved example with the expert-paired related problem from MathNet-RAG, again together with its official solution. Table 5: Retrieval-Augmented Problem Solving Accuracy (%, ↑) on MathNet-RAG (35 problems).

Citation

@misc{alshammari2026mathnet,
  title={MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval},
  author={Shaden Alshammari et al.},
  year={2026},
  note={arXiv:2604.18584}
}

• arXiv: 2604.18584