Mathnet Retrieve Eval

mathnet-retrieve-eval

MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval — Shaden Alshammari et al. (arXiv:2604.18584, 2026)

What this evaluates

Probes a model's ability to retrieve mathematically equivalent problems from a large corpus using embeddings. It measures whether retrieval systems can recognize structural and symbolic invariance rather than relying on superficial lexical overlap.

Datasets

• MathNet-Retrieve — total 10000; splits: test (10000); repo https://github.com/ShadeAlsha/MathNet

Metrics

• Recall@k (primary) — range: percent
  • Measures whether any of the top-k retrieved problems correspond to a “correct” match from the equivalent versions of each anchor problem. Reported for k=1 and k=5.

Input / output format

Input: Embedding of the anchor problem statement.

Output: Ranked list of retrieved problem statements.

Scoring recipe

retrieved = retrieve_top_k(anchor_embedding, corpus, k=k)
is_hit = any(retrieved_problem in correct_equivalent_set for retrieved_problem in retrieved)
recall_at_k = sum(is_hit) / total_anchors

Common pitfalls

• Embeddings often match on superficial keywords (e.g., 'triangle') rather than mathematical structure, leading to low Recall@1 despite high Recall@10.
• Uses cosine similarity over embeddings, which may not capture symbolic invariance.
• Hard negatives (near misses) are explicitly constructed to test fine-grained distinctions, but models struggle to separate them.

Evidence (verbatim from paper)

The primary evaluation metric for our retrieval task is Recall@k, which measures whether any of the top-k retrieved problems correspond to a “correct” match from our equivalent versions of each problem. We report Recall@1 and Recall@5.

Citation

@misc{alshammari2026mathnet,
  title={MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval},
  author={Shaden Alshammari et al.},
  year={2026},
  note={arXiv:2604.18584}
}

• arXiv: 2604.18584