Long Cot Reasoning Eval

long-cot-reasoning-eval

Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning — Yan et al. (2026) (arXiv:2601.09088, 2026)

What this evaluates

This evaluation probes a model's ability to perform complex, multi-step reasoning across mathematics, coding, and scientific domains. It specifically measures the capacity to generate long chain-of-thought traces and produce correct final answers or executable code under strict generation constraints.

Datasets

• AIME24 — total 30; splits: test (30)
• AIME25 — total 30; splits: test (30)
• GPQA Diamond — total 198; splits: test (198)
• LiveCodeBench v5 — total ?; splits: test (-1)
• LiveCodeBench v6 — total ?; splits: test (-1)

Metrics

• average accuracy (primary) — range: percent
  • Average accuracy across 64 sampled responses per question. For AIME and GPQA-D, this is exact-match accuracy on the final answer. For LiveCodeBench, it measures executable correctness and test-output prediction accuracy.

Input / output format

Input: Natural language problem statement or coding/scientific question.

Output: Autoregressive chain-of-thought reasoning trace followed by a final answer or code block.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if exact_match(pred, gold): # or check executable correctness for code
            correct += 1
    return (correct / len(golds)) * 100

# Protocol:
# 1. Sample 64 responses per question (temp=1.0, top_p=1.0)
# 2. Max tokens: 102400 for AIME, 81920 for LCB/GPQA-D
# 3. Extract final answer/code from each response
# 4. Compare against gold using exact match or execution
# 5. Average accuracy across all questions

Common pitfalls

• Failing to sample multiple responses (64) per question, which leads to unstable accuracy estimates for stochastic models.
• Using incorrect max token limits: AIME requires 102,400 tokens while LCB/GPQA-D require 81,920 tokens.
• Confusing LiveCodeBench temporal splits (v5 vs v6) which have different problem collections and contamination controls.

Evidence (verbatim from paper)

For every benchmark, we sampled 64 responses per question and reported the average accuracy to ensure reliable and stable evaluation results. Given the extreme difficulty of AIME24 and AIME25, we set the maximum generation length to 102,400 tokens; For LiveCodeBench and GPQA-D, the limit was set to 81,920 tokens.

Citation

@misc{yan2026dasd,
  title={Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning},
  author={Yan et al. (2026)},
  year={2026},
  note={arXiv:2601.09088}
}

• arXiv: 2601.09088