Gsm8k Eval

gsm8k-eval

Training Verifiers to Solve Math Word Problems — Cobbe et al. (2021) (arXiv:2110.14168, 2021)

What this evaluates

Evaluates a model's ability to perform multi-step arithmetic reasoning by generating natural language solutions to grade school math word problems and verifying their correctness.

Datasets

• GSM8K — total ?; splits: train (-1), test (-1)

Metrics

• solve rate (primary) — range: percent
  • The percentage of test problems for which the model's generated final answer exactly matches the ground truth answer. Evaluated as test@1 (single low-temperature sample) or test@N (best-of-N samples).

Input / output format

Input: A natural language grade school math word problem.

Output: A step-by-step natural language solution followed by a final numerical answer.

Scoring recipe

def compute_solve_rate(predictions, gold_answers, n_samples=1):
    correct = 0
    for i in range(0, len(predictions), n_samples):
        batch = predictions[i:i+n_samples]
        gold = gold_answers[i // n_samples]
        final_answers = [extract_final_answer(completion) for completion in batch]
        if gold in final_answers:
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Models may reach the correct final answer using flawed reasoning, leading to false positives in verification.
• Overfitting to the training set causes the model to become overconfident and reduces solution space coverage at test time.
• Temperature selection critically impacts the trade-off between solution diversity (coverage) and calibration.

Evidence (verbatim from paper)

At test time, we judge performance by autoregressively sampling a single low temperature solution and checking whether the final answer is correct. ... We use test@N to denote the percentage of problems solved correctly at least once when allowing the model to make N separate guesses for each problem. ... estimate that a model with 10^16 parameters would be required to reach an 80% solve rate, when using the full GSM8K training set.

Citation

@misc{cobbe2021trainingverifiers,
  title={Training Verifiers to Solve Math Word Problems},
  author={Cobbe et al. (2021)},
  year={2021},
  note={arXiv:2110.14168}
}

• arXiv: 2110.14168