Omni Math Eval

omni-math-eval

Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models — Gao et al. (2024) (arXiv:2410.07985, 2024)

What this evaluates

Evaluates large language models on rigorous, Olympiad-level mathematical reasoning across diverse domains and difficulty levels. It probes the model's ability to perform complex logical deduction, multi-step problem solving, and handle non-standard answer formats without relying on trivial or non-mathematical content.

Datasets

• Omni-MATH — total 4428; splits: test (4428); repo https://github.com/KbsdJames/Omni-MATH

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of problems where the model's generated solution is judged consistent with the reference answer. Calculated as (number of correctly solved problems) / (total number of problems).

Input / output format

Input: Mathematical problem statement (typically in LaTeX or text) and the corresponding reference answer.

Output: Model-generated solution or final answer.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        # LLM judge compares pred.solution with gold.answer
        is_consistent = llm_judge_consistency_check(pred.solution, gold.answer)
        if is_consistent:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Diverse final answer formats make fixed rule-based evaluation unreliable, necessitating LLM-based consistency checks.
• Model outputs may contain correct reasoning but fail due to formatting issues or require additional steps to extract the final answer for evaluation.

Evidence (verbatim from paper)

We employ cross-validation to enhance the robustness of our findings, yielding an accuracy rate of 92.7%.

Citation

@misc{gao2024omnimath,
  title={Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models},
  author={Gao et al. (2024)},
  year={2024},
  note={arXiv:2410.07985}
}

• arXiv: 2410.07985