Extra Cot Eval

extra-cot-eval

Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression — Yuntian Tang et al. (arXiv:2602.08324, 2026)

What this evaluates

Evaluates the ability of large language models to generate mathematically reasoned chain-of-thought outputs that are compressed to a target token budget while preserving logical fidelity and answer accuracy.

Datasets

• GSM8K — total ?; splits: test (-1)
• MATH-500 — total ?; splits: test (-1)
• AMC2023 — total ?; splits: test (-1)

Metrics

• Acc@all (primary) — range: percent
  • Accuracy computed over the entire test set by comparing the model's extracted final answer to the ground truth label.
• ActRatio — range: [0, 1]
  • The realized compression ratio aggregated over the dataset, calculated as the mean of compressed-to-original token ratios strictly within ... blocks, averaged over parsable outputs.

Input / output format

Input: A mathematical reasoning problem or question.

Output: A compressed chain-of-thought reasoning trace enclosed in ... tags, followed by the final answer.

Scoring recipe

def score(predictions, golds, original_tokens):
    correct = 0
    act_ratios = []
    for pred, gold, orig in zip(predictions, golds, original_tokens):
        if pred.strip() == gold.strip():
            correct += 1
        compressed_tokens = count_tokens_in_think_block(pred)
        original_think_tokens = count_tokens_in_think_block(orig)
        if original_think_tokens > 0:
            act_ratios.append(compressed_tokens / original_think_tokens)
    accuracy = correct / len(golds)
    act_ratio = sum(act_ratios) / len(act_ratios) if act_ratios else 0
    return accuracy, act_ratio

Common pitfalls

• Token counts and ActRatio are computed exclusively within ... blocks, excluding the final answer tokens.
• ActRatio is averaged only over outputs that are successfully parsed for blocks; unparsable outputs are excluded from the ratio calculation.
• Models may exhibit 'control collapse' where they ignore the target compression ratio to prioritize accuracy, leading to ActRatio significantly exceeding the target.

Evidence (verbatim from paper)

Evaluation protocol. We evaluate all methods across five target compression ratios: γ∈{0.2,0.4,0.6,0.8,1.0}, and the mode. Our evaluation centers on two primary metrics: accuracy and compression efficiency. Accuracy (Acc@all) is computed over the entire test set. To measure compression efficiency, we report the Actual Ratio (ActRatio), which is the realized compression ratio aggregated over the dataset. Crucially, all token counts and ratios are computed using think-only accounting, measuring tokens strictly within ... blocks. Consequently, ActRatio and its underlying token counts are averaged over parsable outputs.

Citation

@misc{tang2026extracot,
  title={Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression},
  author={Yuntian Tang et al.},
  year={2026},
  note={arXiv:2602.08324}
}

• arXiv: 2602.08324