Phi 4 Reasoning Eval

phi-4-reasoning-eval

Phi-4-reasoning Technical Report — Marah Abdin et al. (arXiv:2504.21318, 2025)

What this evaluates

Evaluates large language models on reasoning-specific capabilities including mathematics, scientific QA, coding, algorithmic planning, and spatial reasoning. It probes the model's ability to generate step-by-step solution traces and produce correct final answers under varying decoding temperatures and run counts.

Datasets

• AIME — total ?; splits: test (-1)
• GPQA Diamond — total ?; splits: test (-1)
• OmniMATH — total 4000; splits: test (-1)
• LiveCodeBench — total ?; splits: test (-1)
• Codeforces — total 143; splits: test (-1)

Metrics

• pass@1 accuracy (primary) — range: [0, 1]
  • Fraction of problems solved correctly across N independent generation runs. Computed as the number of problems with at least one correct run divided by the total number of problems, averaged over 5 or 50 runs for statistical robustness.
• Elo rating — range: other
  • Statistical measure of relative skill based on pairwise comparisons across 10 independent submission attempts per problem.

Input / output format

Input: Problem statement preceded by a system message and a CoT template explicitly requiring step-by-step thinking (except for o1 which uses a plain non-CoT template due to policy refusals).

Output: A step-by-step reasoning trace followed by a final answer.

Scoring recipe

def compute_pass_at_1(predictions, gold, num_runs=50):
    correct = 0
    for i, gold_ans in enumerate(gold):
        for run in range(num_runs):
            if predictions[run][i] == gold_ans:
                correct += 1
                break
    return correct / len(gold)

Common pitfalls

• Relying on a single generation run for small benchmarks (e.g., AIME 2025 with 30 problems) causes high variance (5-10 pp swings) and unreliable model comparisons.
• Using CoT prompts for models like o1 triggers policy refusals; a plain non-CoT template is required for those specific models.
• Ignoring token-length tradeoffs, as reasoning models often produce longer traces that do not always correlate with higher accuracy on complex tasks.

Evidence (verbatim from paper)

For all models including ours and those in the OpenAI and DeepSeek family, average-of-5 results from two independent runs can differ significantly by up to 5-10 percentage points on AIME 2025, thus making side-by-side comparison of models unreliable. To mitigate this issue and increase the statistical robustness of results, we report pass@1 accuracy averaged over 50 independent runs in Table 1 and Figure 1.

Citation

@misc{abdin2025phi4reasoning,
  title={Phi-4-reasoning Technical Report},
  author={Marah Abdin et al.},
  year={2025},
  note={arXiv:2504.21318}
}

• arXiv: 2504.21318