Sbsc Math Olympiad Eval

sbsc-math-olympiad-eval

SBSC: Step-By-Step Coding for Improving Mathematical Olympiad Performance — Kunal Singh et al. (2025) (arXiv:2502.16666, 2025)

What this evaluates

Evaluates LLMs' ability to solve complex, Olympiad-level mathematics problems across Algebra, Combinatorics, Number Theory, and Geometry. It specifically probes multi-turn code generation and execution feedback for iterative problem decomposition and constraint handling.

Datasets

• AIME — total 330; splits: test (330)
• AMC-12 — total 475; splits: test (475)
• MathOdyssey — total 158; splits: test (158)
• OlympiadBench — total 504; splits: test (504)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy: 1 if the model's final extracted integer answer matches the gold integer answer, 0 otherwise. Averaged across all test instances.

Input / output format

Input: LaTeX-formatted math competition problems, modified to require integer answers. Evaluated in a 4-shot setting with carefully selected exemplars that all baseline methods solved correctly.

Output: Multi-turn code generation and execution traces, culminating in a final integer answer. For self-consistency decoding, 7 samples are generated and the majority vote is taken.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_int = extract_integer_from_text(pred)
        if pred_int == gold:
            correct += 1
    return correct / len(golds)

Common pitfalls

• AMC-12 questions are originally multiple-choice; the paper removes answer choices and forces integer answers, altering the standard evaluation format.
• MathOdyssey and OlympiadBench questions are post-processed via GPT-4o to append integer-answer prompts, which may not reflect the original benchmark's intended format.
• The 4-shot exemplars are specifically curated to be correctly solved by all methods, potentially inflating performance compared to random or zero-shot settings.

Evidence (verbatim from paper)

Note: we experimentally observe that for n > 4, there is insignificant increase in accuracy for TIR-ToRA so we set n=4 for TIR-ToRA during SC decoding.

Citation

@misc{singh2025sbsc,
  title={SBSC: Step-By-Step Coding for Improving Mathematical Olympiad Performance},
  author={Kunal Singh et al. (2025)},
  year={2025},
  note={arXiv:2502.16666}
}

• arXiv: 2502.16666