Gsm8k Eval

gsm8k-eval — Grade-school math word problems

Source: Cobbe et al., "Training Verifiers to Solve Math Word Problems" (arXiv:2110.14168).

What this benchmark tests

Multi-step arithmetic reasoning over 2–8-step word problems. Solutions are natural language with a final numeric answer. Designed to be solvable by a "bright middle school student" but to expose LMs' shaky chain-of-thought arithmetic.

When to invoke this skill

• User says "test on GSM8K" / "evaluate math reasoning" / "math word problems"
• User asks for a single-number sanity check on basic LM math
• User mentions verifier-style approaches (test@N), self-consistency, MathPrompter, etc. — they all use this dataset

Dataset structure

Each example has:

{
  "question": "Natalia sold clips to 48 of her friends in April...",
  "answer": "Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips in April and May.\n#### 72"
}

• The chain-of-thought is in plain text, with <<expr=val>> calculator annotations.
• The final numeric answer is everything after ####.

Splits (HF gsm8k, config main or socratic):

• train: 7,473 problems (use for finetuning, NOT eval)
• test: 1,319 problems (the only thing you should report on)

Evaluation protocol

1. Input prompt: the question text. The paper's recipe is direct: ask the model for a step-by-step solution ending with #### <number>. Modern recipes use 8-shot CoT prompts (Wei et al. 2022) for reproducibility.
2. Sampling:
   • test@1: temperature 0, single sample → check final answer.
   • test@N: temperature 0.7, N samples → check whether any is correct (pass@N).
   • maj@N: temperature 0.7, N samples → take majority vote on extracted answer.
3. Answer extraction: regex the model's final line for ####\s*(-?[\d,]+(?:\.\d+)?). Strip commas. Compare numerically (not as string) to the gold final answer (also extracted via the same regex from the gold answer).
4. Metric: exact_match_accuracy = #correct / #total. Report 2–4 decimal places.

Reference scoring

import re
def extract_answer(text: str) -> float | None:
    m = re.search(r"####\s*(-?[\d,]+(?:\.\d+)?)", text)
    return float(m.group(1).replace(",", "")) if m else None

def gsm8k_em(pred: str, gold: str) -> bool:
    p, g = extract_answer(pred), extract_answer(gold)
    return p is not None and g is not None and abs(p - g) < 1e-6

# usage
from datasets import load_dataset
ds = load_dataset("gsm8k", "main", split="test")
correct = sum(gsm8k_em(your_model_output(ex["question"]), ex["answer"]) for ex in ds)
print(f"acc = {correct/len(ds):.4f}")

Reporting format

Always state exact-match accuracy + which protocol:

• gsm8k acc (test@1, 0-shot CoT) = 0.812
• gsm8k acc (test@1, 8-shot CoT) = 0.873
• gsm8k maj@64 (T=0.7) = 0.926

Don'ts

• Don't compare different sampling configs as if they were the same metric — maj@64 ≠ test@1.
• Don't report on train. The 1319 in test is the only legitimate number.
• Don't string-compare the final answer; "1,000" vs "1000" vs "1000.0" must all match.
• Don't forget that some gold answers are negative; the regex must capture leading -.

Citation

@article{cobbe2021gsm8k,
  title={Training Verifiers to Solve Math Word Problems},
  author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and others},
  journal={arXiv:2110.14168}, year={2021}
}