Longcot Eval

longcot-eval

LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning — Motwani et al. (2026) (arXiv:2604.14140, 2026)

What this evaluates

Probes a model's ability to maintain coherent state, plan, and execute multi-step reasoning over long, interdependent chains of thought spanning tens to hundreds of thousands of tokens across domains like chemistry, mathematics, and chess.

Datasets

• LongCoT — total 2500; splits: test (2500)

Metrics

• accuracy (primary) — range: percent
  • Percentage of questions where the final answer matches the expected answer after sequential verification (RegEx, flexible RegEx, LLM extraction, or manual check).

Input / output format

Input: Single-shot problem descriptions requiring navigation of complex dependency graphs (DAGs), with reasoning budgets up to provider limits (e.g., 128K tokens).

Output: Chain-of-thought reasoning trace followed by a final answer in the expected format.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        # Sequential verification pipeline
        answer = extract_answer(pred) # RegEx -> flexible RegEx -> LLM fallback
        if answer == g: # Manual verification fallback if needed
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Assuming pass@k or self-consistency can be easily scaled; high token costs prevent these experiments.
• Expecting domain-specific knowledge to drive performance; success relies on long-horizon state maintenance and planning, not deep domain expertise.
• Assuming accuracy degradation is solely due to context window saturation; the benchmark shows performance drops sharply as DAG complexity increases, independent of context limits.

Evidence (verbatim from paper)

Final answers are verified through sequential checks: RegEx on expected format, flexible RegEx on full responses if needed, and LLM-based extraction (GPT-5-mini) as a fallback. These answers are then manually verified for correctness. We find that performance is uniformly low, with GPT 5.2 achieving the highest accuracy of 9.83% followed by Gemini 3 Pro (6.08%) and Grok 4.1 Fast Reasoning (2.04%).

Citation

@misc{motwani2026longcot,
  title={LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning},
  author={Motwani et al. (2026)},
  year={2026},
  note={arXiv:2604.14140}
}

• arXiv: 2604.14140