Code Reasoning Eval

code-reasoning-eval

How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data — Huang et al. (2026) (arXiv:2604.14164, 2026)

What this evaluates

This evaluation probes a model's ability to solve programming contest problems and reason through mathematics and science questions. It measures how well fine-tuning on synthesized data preserves or enhances in-domain coding capabilities while maintaining out-of-domain generalization across multiple reasoning benchmarks.

Datasets

• LiveCodeBench-V5 — total ?; splits: test (-1)
• LiveCodeBench-V6 — total ?; splits: test (-1)
• LiveCodeBench-Pro — total ?; splits: test (-1)
• OJBench — total ?; splits: test (-1)
• AIME-2024 — total ?; splits: test (-1)
• AIME-2025 — total ?; splits: test (-1)
• OlympiadBench — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)

Metrics

• pass@1 (primary) — range: percent
  • Computed as the average pass rate across multiple independent generation runs. For code tasks, a run passes if the generated code solves the problem (e.g., passes hidden test cases). For math/science, it checks exact match or logical equivalence against the gold answer.

Input / output format

Input: Text-only programming contest questions, mathematics problems, and science QA prompts.

Output: Model-generated reasoning traces and final answers/code solutions.

Scoring recipe

def compute_pass_at_1(predictions_per_run, gold_answers):
    run_pass_rates = []
    for run_preds in predictions_per_run:
        correct = sum(1 for p in run_preds if check_solution(p, gold_answers))
        run_pass_rates.append(correct / len(run_preds))
    return sum(run_pass_rates) / len(run_pass_rates) * 100

Common pitfalls

• pass@1 must be averaged over multiple independent generation runs rather than calculated from a single greedy or sampled output.
• For multimodal evaluation datasets, only the text-only components are used; visual inputs are stripped before evaluation.
• Training on fully teacher-generated data often causes performance degradation on the student model, contrary to naive distillation assumptions.

Evidence (verbatim from paper)

All evaluations were conducted on the OpenCompass [opencompass] platform, where pass@1 is computed as the average of results from multiple independent runs.

Citation

@misc{huang2026tessy,
  title={How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data},
  author={Huang et al. (2026)},
  year={2026},
  note={arXiv:2604.14164}
}

• arXiv: 2604.14164