qhjqhj00 frontiermath-eval
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI — Glazer et al. (2024) (arXiv:2411.04872, 2024)
What this evaluates
Evaluates advanced mathematical reasoning and experimental problem-solving. It tests whether models can iteratively write and execute Python code to verify hypotheses, refine strategies, and derive correct solutions to expert-level, unsolved math problems.
Datasets
- FrontierMath — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: [0, 1]- Proportion of problems for which the model submits a correctly formatted final answer that matches the ground truth. Calculated as (number of solved problems) / (total problems). Reported as mean accuracy across multiple runs.
Input / output format
Input: A mathematical problem statement requiring advanced reasoning.
Output: A self-contained Python code block containing the final answer, preceded by the exact marker comment # This is the final answer, and saved using Python's pickle module.
Scoring recipe
def score(predictions, golds):
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, golds):
if pred is None:
continue
if '# This is the final answer' not in pred:
continue
if extract_final_answer(pred) == gold:
correct += 1
return correct / total if total > 0 else 0
Common pitfalls
- Low overall success rate (<2%) means model rankings are highly unstable and sensitive to single successes.
- A correct final answer does not guarantee valid mathematical reasoning, as some problems can be solved via simulation or guessing without deeper understanding.
- Models forced by the 10,000-token limit may submit incorrect or unformatted answers, skewing accuracy if not handled uniformly.
Evidence (verbatim from paper)
Figure 6: Performance of leading language models on FrontierMath based on mean accuracy across 8 runs. All models show consistently poor performance, with even the best models solving less than 2% of problems in each run on average.
Citation
@misc{glazer2024frontiermath,
title={FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI},
author={Glazer et al. (2024)},
year={2024},
note={arXiv:2411.04872}
}
- arXiv: 2411.04872