Imo Shortlist Eval

imo-shortlist-eval

Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics — Mahdavi et al. (2025) (arXiv:2504.01995, 2025)

What this evaluates

This benchmark probes an LLM's ability to produce logically sound, step-by-step mathematical reasoning for Olympiad-level problems. It specifically measures the gap between achieving the correct final answer and maintaining rigorous, fallacy-free solution processes.

Datasets

• IMO shortlist problems (2009-2023) — total 455; splits: test (455)

Metrics

• Final Answer Accuracy (%) (primary) — range: percent
  • Percentage of problems where the model's final numerical or symbolic answer matches the ground truth, regardless of the reasoning steps.
• Correct|Correct Final Answer (%) (primary) — range: percent
  • Conditional probability of a fully correct solution (free of logical fallacies) given that the final answer is correct.
• Correct (%) — range: percent
  • Human-annotated percentage of solutions deemed fully correct based on logical soundness and completeness.
• Partially Correct (%) — range: percent
  • Human-annotated percentage of solutions with some valid steps but containing logical gaps or minor errors.
• Incorrect (%) — range: percent
  • Human-annotated percentage of solutions with fundamental errors or completely wrong approaches.

Input / output format

Input: Olympiad-level mathematics problems from the IMO shortlists (2009-2023), spanning algebra, combinatorics, geometry, and number theory.

Output: Step-by-step mathematical solutions concluding with a final answer.

Scoring recipe

def evaluate(predictions, gold_answers, human_labels):
    # 1. Final Answer Accuracy
    correct_answers = sum(1 for p, g in zip(predictions, gold_answers) if p.final_answer == g)
    final_acc = correct_answers / len(predictions)

    # 2. Correct|Correct Final Answer
    correct_answer_indices = [i for i, p in enumerate(predictions) if p.final_answer == gold_answers[i]]
    if not correct_answer_indices:
        correct_given_acc = 0.0
    else:
        correct_solutions = sum(1 for i in correct_answer_indices if human_labels[i].solution_correct)
        correct_given_acc = correct_solutions / len(correct_answer_indices)

    # 3. Fallacy Frequencies
    fallacy_counts = {f: 0 for f in human_labels.fallacy_types}
    for label in human_labels:
        for f in label.fallacies:
            fallacy_counts[f] += 1
    return final_acc, correct_given_acc, fallacy_counts

Common pitfalls

• Relying solely on final answer accuracy grossly overestimates reasoning abilities, as models often use heuristics or trial-and-error to guess answers while containing logical fallacies.
• Human evaluation is strictly required to assess solution soundness and identify specific fallacies (e.g., Proof by Example, Inventing Wrong Facts), which automated exact-match checks cannot capture.
• Problems without explicit final answers require logical proofs, which significantly increases the frequency of fallacies like Proposal Without Verification and Begging the Question.

Evidence (verbatim from paper)

Final Answer Accuracy denotes the percentage of correct final answers, whereas Correct|Correct Final Answer represents the percentage of fully correct solutions among instances where the final answer is correct.

Citation

@misc{mahdavi2025brainsvsbytes,
  title={Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics},
  author={Mahdavi et al. (2025)},
  year={2025},
  note={arXiv:2504.01995}
}

• arXiv: 2504.01995