Math Best Of N Eval

math-best-of-n-eval

Let's Verify Step by Step — Lightman et al. (2023) (arXiv:2305.20050, 2023)

What this evaluates

Evaluates the reliability of reward models for mathematical reasoning by selecting the best solution from a set of sampled candidates using best-of-N search, comparing outcome versus process supervision.

Datasets

• MATH — total ?; splits: train (12000), test (500)

Metrics

• fraction_correct (primary) — range: [0, 1]
  • Sample N solutions per problem from a fixed generator. Rank them using the reward model. Select the top-ranked solution and check if its final answer matches the ground truth. Report the fraction of problems where the selected solution is correct.

Input / output format

Input: Text-based mathematical problems from the MATH dataset.

Output: Newline-delimited step-by-step solutions ending with a final answer.

Scoring recipe

correct_count = 0
for problem in test_set:
    solutions = generator.sample(problem, n=N)
    scores = [reward_model.score(sol) for sol in solutions]
    best_sol = solutions[argmax(scores)]
    if check_final_answer(best_sol) == problem.gold_answer:
        correct_count += 1
return correct_count / len(test_set)

Common pitfalls

• Automatic grading of final answers can produce false positives when solutions reach the correct answer via incorrect reasoning.
• PRM scores are computed as the product of per-step correctness probabilities, making them sensitive to step boundaries and tokenization artifacts.
• Neutral labels in process supervision are deferred at test time, meaning PRM scoring behavior depends on how ambiguity is resolved.

Evidence (verbatim from paper)

We evaluate a reward model by its ability to perform best-of-N search over uniformly sampled solutions from the generator. For each test problem we select the solution ranked highest by the reward model, automatically grade it based on its final answer, and report the fraction that are correct.

Citation

@misc{lightman2023letsverify,
  title={Let's Verify Step by Step},
  author={Lightman et al. (2023)},
  year={2023},
  note={arXiv:2305.20050}
}

• arXiv: 2305.20050