Implicit Cot Eval

implicit-cot-eval

From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step — Deng et al. (2024) (arXiv:2405.14838, 2024)

What this evaluates

Evaluates a model's ability to perform arithmetic and grade-school math reasoning without generating explicit intermediate chain-of-thought steps. It measures both the exact-match accuracy of the final answer and the inference speed relative to a no-CoT baseline.

Datasets

• Multi-digit multiplication — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1); HF gsm8k

Metrics

• Accuracy (primary) — range: [0, 1]
  • Exact match accuracy of producing the final answer. Calculated as the number of correctly predicted final answers divided by the total number of test instances.
• Speed — range: other
  • Inference throughput measured in examples per second on an Nvidia H100 GPU with batch size 1, normalized by the speed of the corresponding No CoT baseline model.

Input / output format

Input: Arithmetic problems (e.g., K-by-K multiplication equations or grade-school math word problems).

Output: A single final numerical answer or solution string, without intermediate reasoning steps.

Scoring recipe

def compute_metrics(predictions, golds, model, baseline_model):
    # Accuracy
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
    accuracy = correct / len(golds)
    # Speed (normalized)
    speed_raw = count_examples_per_second(model, batch_size=1, device='H100')
    speed_baseline = count_examples_per_second(baseline_model, batch_size=1, device='H100')
    speed_normalized = speed_raw / speed_baseline
    return {'accuracy': accuracy, 'speed_normalized': speed_normalized}

Common pitfalls

• Implicit CoT models do not output reasoning traces, so step-level evaluation metrics are inapplicable.
• Speed comparisons must be normalized to the No-CoT baseline on identical hardware and batch size; raw examples/second are not comparable across setups.
• GSM8K results use augmented training data from Deng et al., which may differ from standard splits or prompts.

Evidence (verbatim from paper)

Accuracy (Acc in the table) measures the exact match accuracy of producing the final answer. Speed measures the number of examples per second during inference using a batch size of 1, normalized by the speed of the corresponding No CoT model.

Citation

@misc{deng2024implicitcot,
  title={From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step},
  author={Deng et al. (2024)},
  year={2024},
  note={arXiv:2405.14838}
}

• arXiv: 2405.14838