Booststep Eval

booststep-eval

BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning — Zhang et al. (2025) (arXiv:2501.03226, 2025)

What this evaluates

This evaluation probes the mathematical reasoning capability of large language models, specifically focusing on single-step reasoning and the effectiveness of step-aligned in-context learning. It measures how well models can solve challenging math problems across text and multi-modal domains when provided with fine-grained, step-level examples.

Datasets

• MATH500 — total 500; splits: test (-1)
• AQuA — total ?; splits: test (-1)
• OlympiadBench-TO — total ?; splits: test (-1)
• MATHBench — total ?; splits: test (-1)
• AMC-10 — total ?; splits: test (-1)
• AMC-12 — total ?; splits: test (-1)
• MathVision — total ?; splits: test (-1)
• MathVerse — total ?; splits: test (-1)
• AIME — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of problems where the model's final predicted answer exactly matches the ground truth answer. Calculated as (number of correct predictions / total number of problems) * 100.

Input / output format

Input: A mathematical problem (text or image-based for multi-modal benchmarks) accompanied by a prompt containing step-aligned in-context examples retrieved via TF-IDF cosine similarity. For tree search variants, multiple reasoning candidates are generated.

Output: A step-by-step reasoning chain followed by a final answer. The model may generate multiple candidates during tree search, from which a verifier selects the best path.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if extract_final_answer(pred) == gold:
        correct += 1
accuracy = (correct / len(predictions)) * 100

Common pitfalls

• Using grammatical segmentation (e.g., splitting by periods) instead of reasoning-content-based step splitting for the example problem bank, which significantly degrades performance.
• Assuming problem-level few-shot learning is universally superior; step-level ICL can fail or underperform if the retrieval strategy lacks the 'first-try' mechanism or if example similarity is too low.
• Forgetting to adjust the temperature parameter: it must be set to 0 for standard generation but 0.3 when using step-level tree search to allow random sampling.

Evidence (verbatim from paper)

Considering that the AIME consists of only 30 questions each year, making the results prone to fluctuations, we evaluated the questions three times annually and reported the average accuracy.

Citation

@misc{zhang2025booststep,
  title={BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2501.03226}
}

• arXiv: 2501.03226