Spice Reasoning Eval

spice-reasoning-eval

SPICE: Self-Play In Corpus Environments Improves Reasoning — Liu et al. (2025) (arXiv:2510.24684, 2025)

What this evaluates

This evaluation probes a model's ability to solve challenging mathematical and general reasoning tasks, both from standard benchmarks and document-grounded self-play generated questions. It measures how well the model can extract information, perform multi-step logical deduction, and produce verifiable answers across diverse academic and competition-level datasets.

Datasets

• MATH-500 — total ?; splits: test (-1)
• OlympiadBench — total ?; splits: test (-1)
• Minerva Math — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• AMC — total ?; splits: test (-1)
• AIME'24 — total ?; splits: test (-1)
• AIME'25 — total ?; splits: test (-1)
• SuperGPQA — total ?; splits: test (-1)
• GPQA-Diamond — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)
• BBEH — total ?; splits: test (-1)

Metrics

• pass rate (primary) — range: percent
  • Percentage of correctly answered questions out of the total evaluated. Correctness is determined by checking answer equivalence using GPT-4o rather than exact string matching.

Input / output format

Input: Standard benchmark questions or document-grounded questions, formatted with training-consistent prompts following Ma et al. (2025).

Output: Chain-of-thought reasoning steps followed by a final answer.

Scoring recipe

def compute_pass_rate(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if gpt4o_check_equivalence(pred, gold):
            correct += 1
    return (correct / len(predictions)) * 100
# Note: AIME'24 and AIME'25 require averaging over 32 sampling runs instead of greedy decoding.

Common pitfalls

• AIME'24 and AIME'25 require averaging results over 32 sampling runs, not greedy decoding.
• Answer correctness is evaluated via GPT-4o equivalence checking, not exact string or regex matching.
• Evaluation prompts must strictly follow the training-consistent templates from Ma et al. (2025) to ensure fair comparison with the training setup.

Evidence (verbatim from paper)

Figure 3: Reasoner pass rates when evaluating SPICE checkpoints at steps 200-640 against a fixed step-200 checkpoint on 128 documents. (a) Fixed Reasoner: Pass rate decreases from 55% to 35% as later Challenger checkpoints generate harder questions. (b) Fixed Challenger: Pass rate increases from 55% to 85% as later Reasoner checkpoints improve at solving questions.

Citation

@misc{liu2025spice,
  title={SPICE: Self-Play In Corpus Environments Improves Reasoning},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.24684}
}

• arXiv: 2510.24684