Neural Cot Search Eval

neural-cot-search-eval

Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models — Ling et al. (2026) (arXiv:2601.11340, 2026)

What this evaluates

Evaluates large language models' ability to perform multi-step reasoning across diverse domains including mathematics, commonsense, and expert knowledge. It specifically probes the model's capacity to generate accurate solutions while minimizing computational cost (token usage) through dynamic reasoning path search.

Datasets

• AMC23 — total ?; splits: test (-1)
• ARC-C — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)

Metrics

• Accuracy — range: [0, 1]
  • Proportion of generated answers that exactly match the ground truth final answer.
• Length — range: tokens
  • Average number of tokens generated per query.
• Efficiency Metric ($\eta$) (primary) — range: other
  • Composite score: $\eta = (\frac{\mathbb{E}[A_{model}]}{\mathbb{E}[A_{baseline}]})^2 \times \frac{\mathbb{E}[L_{baseline}]}{\mathbb{E}[L_{model}]}$. It squares the accuracy ratio to prioritize correctness, then multiplies by the token reduction ratio. Values >1 indicate better reasoning density than the baseline.

Input / output format

Input: Natural language question or problem statement, appended with the prompt: Please reason step by step, and put your final answer within \boxed{}

Output: Step-by-step reasoning text followed by the final answer enclosed in \boxed{}.

Scoring recipe

def compute_metrics(model_outputs, gold_answers, baseline_outputs):
    # Compute Accuracy
    acc_model = sum(1 for p, g in zip(model_outputs, gold_answers) if p.strip() == g.strip()) / len(gold_answers)
    acc_base = sum(1 for p, g in zip(baseline_outputs, gold_answers) if p.strip() == g.strip()) / len(gold_answers)
    
    # Compute Length (token count)
    len_model = sum(len(p.split()) for p in model_outputs) / len(model_outputs)
    len_base = sum(len(p.split()) for p in baseline_outputs) / len(baseline_outputs)
    
    # Compute Efficiency Metric
    eta = (acc_model / acc_base) ** 2 * (len_base / len_model)
    return {'accuracy': acc_model, 'length': len_model, 'eta': eta}

Common pitfalls

• The Efficiency Metric ($\eta$) is a relative comparison against a baseline (Original or Mean sampling), not an absolute score. Reporting it without specifying the baseline makes it incomparable.
• Length is measured in tokens, but the paper uses expected values over multiple sampled reasoning paths per query. Single-pass generation will not match the reported averages.
• The prompt enforces a specific \boxed{} format for the final answer. Parsers must extract content inside the box to compute accuracy correctly.

Evidence (verbatim from paper)

We report task-specific Accuracy ($A$) and the average token count ($L$). To quantify the trade-off between performance gains and computational cost, we adopt a composite Efficiency Metric ($\eta$), inspired by previous works on efficient reasoning An et al. (2025); Qu et al. (2025a). This metric places a quadratic emphasis on accuracy, as computational savings are secondary to correctness: [formula] Here, $\pi^{*}$ denotes our search-augmented policy and $\pi$ represents the original model. $A(\cdot)$ measures solution correctness and $L(\cdot)$ denotes sequence length. A value of $\eta>1$ indicates that the method improves the reasoning density and provides more correct reasoning per unit of computation.

Citation

@misc{ling2026neuralchainofthoughtsearch,
  title={Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models},
  author={Ling et al. (2026)},
  year={2026},
  note={arXiv:2601.11340}
}

• arXiv: 2601.11340