qhjqhj00 u-math-eval
U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs — Chernyshev et al. (2024) (arXiv:2412.03205, 2024)
What this evaluates
Evaluates LLMs' ability to solve university-level mathematical problems, both text-based and multimodal. It also includes a meta-evaluation component to assess how well models can judge the correctness of free-form mathematical solutions.
Datasets
- U-MATH — total 1100; splits: test (1100); repo https://github.com/toloka/u-math
Metrics
accuracy(primary) — range: percent- Percentage of problems correctly solved, determined by GPT-4o as-a-judge comparing the model's free-form output to the ground truth.
macro F1-score— range: [0, 1]- Harmonic mean of precision and recall for predicting solution correctness (Yes/No). 'Inconclusive' judgments are treated as incorrect (No).
Input / output format
Input: Mathematical problem statement (text or text + image). For meta-evaluation, the problem statement plus a candidate free-form solution.
Output: A free-form mathematical solution. For meta-evaluation, a single label: 'Yes', 'No', or 'Inconclusive'.
Scoring recipe
def score_umath(predictions, golds, judge):
correct = 0
for pred, gold in zip(predictions, golds):
if judge(pred) == gold: correct += 1
return correct / len(golds) * 100
def score_mu_math(predictions, golds):
tp = fp = fn = 0
for pred, gold in zip(predictions, golds):
if pred == 'Inconclusive': pred = 'No'
pred_correct = (pred == 'Yes')
if pred_correct and gold: tp += 1
elif pred_correct and not gold: fp += 1
elif not pred_correct and gold: fn += 1
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
Common pitfalls
- Text-only models evaluated on visual problems often guess or misinterpret missing images, yielding noisy accuracy.
- 'Inconclusive' judge outputs are strictly treated as incorrect, penalizing conservative models and lowering F1.
- Using LLM-as-a-judge introduces systematic bias and false positive/negative rates that limit evaluation precision.
Evidence (verbatim from paper)
We report accuracy based on GPT-4o-2024-08-06 as-a-judge for our final results, despite it not being the best performing judge — due to the model still residing among the top-ranked judges, being the more conservative one in terms of false positive rate, as well as widely available, leading to easier reproducibility. The judge’s output is also further processed by an extractor model (Qwen2.5 72B is fixed for consistency), prompted to produce a single label — either ‘Yes’, ‘No’ or ‘Inconclusive’. We include ‘Inconclusive’ for cases when judge refuses to evaluate or generation fails; such judgments are treated as incorrect.
Citation
@misc{chernyshev2024umath,
title={U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs},
author={Chernyshev et al. (2024)},
year={2024},
note={arXiv:2412.03205}
}
- arXiv: 2412.03205