Mamut Mir Eval

mamut-mir-eval

MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training — Drechsel et al. (2025) (arXiv:2502.20855, 2025)

What this evaluates

Evaluates mathematical information retrieval capabilities by testing whether models can match natural language names or LaTeX formulas to their corresponding mathematical identities from a candidate pool. It probes the model's ability to learn structural and notational variations in mathematical expressions through pretraining and fine-tuning.

Datasets

• MAMUT-generated datasets (MF, MT, NMF, MFR) — total ?; splits: train (-1), test (-1); repo https://github.com/aieng-lab/math-mutator

Metrics

• nDCG (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank k, measuring ranking quality by assigning higher scores to relevant items appearing higher in the list, normalized by the ideal DCG.
• p@k — range: [0, 1]
  • Precision at rank k, measuring the fraction of relevant items in the top-k retrieved results.
• AP — range: [0, 1]
  • Average Precision, the area under the precision-recall curve, averaging precision at each rank where a relevant item is retrieved.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for binary classification of query-relevance.

Input / output format

Input: A query consisting of either a natural language name or a LaTeX formula, paired with a candidate set of formulas to retrieve from.

Output: A ranked list of candidate formulas or binary relevance scores for each candidate.

Scoring recipe

import math
def compute_metrics(predictions, gold):
    k = 10
    p_at_k = len(gold & set(predictions[:k])) / k
    dcg = sum(1 / math.log2(i+2) for i, c in enumerate(predictions) if c in gold)
    idcg = sum(1 / math.log2(i+2) for i in range(min(len(gold), k)))
    ndcg = dcg / idcg if idcg > 0 else 0
    ap = sum(len(gold & set(predictions[:i+1])) / (i+1) for i, c in enumerate(predictions) if c in gold) / len(gold)
    tp = len(gold & set(predictions))
    p = tp / len(predictions) if predictions else 0
    r = tp / len(gold) if gold else 0
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return {'p@k': p_at_k, 'ndcg': ndcg, 'ap': ap, 'f1': f1}

Common pitfalls

• Models pretrained on NMF/MFR during pretraining are excluded from the main comparison to avoid data leakage; only MLM-pretrained models (MF/MT) are fine-tuned on NMF/MFR for fair evaluation.
• Metrics are averaged over five independent fine-tuning runs to ensure robustness, not just a single split or run.

Evidence (verbatim from paper)

We evaluate using both binary classification metrics (precision (P), recall (R), F1) and standard IR ranking metrics: Precisión at $k$ ( $p@k$ ), Average Precision (AP), and nDCG, that are averaged over all test queries, with higher values indicating better performance (Manning, 2009; Radlinski & Craswell, 2010).

Citation

@misc{drechsel2025mamut,
  title={MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training},
  author={Drechsel et al. (2025)},
  year={2025},
  note={arXiv:2502.20855}
}

• arXiv: 2502.20855