Eefsuva Eval

eefsuva-eval

EEFSUVA: A New Mathematical Olympiad Benchmark — Khatibi et al. (2025) (arXiv:2510.01227, 2025)

What this evaluates

Evaluates LLMs' ability to solve nonstandard mathematical Olympiad problems from Eastern European and former Soviet Union competitions. It probes genuine mathematical reasoning and adaptability by testing whether models can solve problems from first principles rather than relying on cached solutions or pattern matching from familiar Western benchmarks.

Datasets

• EEFSUVA — total ?; splits: test (-1)

Metrics

• pass rate (primary) — range: percent
  • Percentage of problems for which the model outputs the correct numerical answer. Each problem is evaluated twice in separate sessions; if either attempt yields the correct answer, the problem is counted as solved.

Input / output format

Input: Mathematical problem statement sent verbatim, without any mention of its source or competition.

Output: Numerical answer (and optional reasoning steps).

Scoring recipe

solved_count = 0
for problem in dataset:
    gold_answer = problem.gold_numerical_answer
    is_solved = False
    for attempt in range(2):
        model_output = llm.generate(problem.prompt, new_session=True)
        if model_output.numerical_answer == gold_answer:
            is_solved = True
            break
    if is_solved:
        solved_count += 1
pass_rate = (solved_count / len(dataset)) * 100

Common pitfalls

• Models may rely on cached solutions from similar IMO problems instead of solving from first principles, leading to incorrect answers despite recognizing the problem type.
• Cross-contamination of context can occur if multiple problems are fed in the same chat session, artificially inflating performance.
• Scoring focuses strictly on the final numerical answer, potentially overlooking valid but differently formatted reasoning or intermediate steps.

Evidence (verbatim from paper)

Once this verification step was completed, we submitted each problem to the LLMs specifically, Gemini 2.5 Pro and Chat GPT 5 Thinking using a brand-new chatbot session for every single problem to prevent any cross contamination of context. After several minutes, the model would output a numerical answer, which we then compared against the verified solution. In order to fully mark a problem as incorrectly solved, we ran each problem in a new chat session twice. ... Overall, with a pass rate of approximately 36%, it is clear that the new advancements still require further development.

Citation

@misc{khatibi2025eefsuva,
  title={EEFSUVA: A New Mathematical Olympiad Benchmark},
  author={Khatibi et al. (2025)},
  year={2025},
  note={arXiv:2510.01227}
}

• arXiv: 2510.01227