Cot Eval

cot-eval

The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning — Kim et al. (2023) (arXiv:2305.14045, 2023)

What this evaluates

Evaluates the zero-shot and few-shot reasoning capabilities of language models, specifically probing their ability to generate step-by-step chain-of-thought rationales and produce correct answers across classification and generation tasks.

Datasets

• BigBench Hard (BBH) — total ?; splits: test (-1)
• P3 — total ?; splits: test (-1)
• MGSM — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels. For classification tasks, the model selects the option with the highest logit probability (Direct) or extracts the answer after a specific indicator phrase following the rationale (CoT).
• exact-match — range: [0, 1]
  • Binary score (1 if prediction exactly matches the gold answer string, 0 otherwise). Used for generation tasks under both Direct and CoT evaluation modes.

Input / output format

Input: Task instruction or prompt followed by the trigger phrase 'Let's think step by step'. For CoT classification evaluation, the prompt includes an indicator phrase '[ANSWER]' inserted between the rationale generation space and the possible options.

Output: Sequential generation of a rationale $r_i^t$ followed by an answer $y_i^t$. Under CoT evaluation, the rationale must be at least 8 tokens long. For classification, the final answer is extracted after '[ANSWER]' or via logit comparison; for generation, the answer is extracted after the indicator phrase.

Scoring recipe

def score(predictions, golds, task_type, eval_mode):
    scores = []
    for pred, gold in zip(predictions, golds):
        if task_type == 'classification':
            if eval_mode == 'direct':
                pred_label = argmax_logit(pred)
            else:
                rationale = pred[:min_tokens(8)]
                pred_label = extract_after(rationale, '[ANSWER]')
            correct = (pred_label == gold)
        else: # generation
            if eval_mode == 'cot':
                pred_text = extract_after(pred, indicator_phrase)
            else:
                pred_text = pred
            correct = (pred_text == gold)
        scores.append(1.0 if correct else 0.0)
    return sum(scores) / len(scores)

Common pitfalls

• Smaller language models frequently fail to generate rationales even with the trigger phrase; the protocol enforces a hard minimum of 8 tokens for the rationale in CoT evaluation.
• Direct and CoT evaluation modes are not interchangeable: CoT fine-tuning can improve CoT scores while degrading Direct evaluation performance on the same model.
• Classification tasks require verbalizer/logit comparison for Direct evaluation, not simple string matching against the gold label.

Evidence (verbatim from paper)

For Direct Evaluation on classification tasks, we follow previous works using verbalizers, choosing the option with the highest probability through comparison of logit values (Schick and Schütze, 2021; Sanh et al., 2021; Ye et al., 2022; Jang et al., 2023), and measure the accuracy. For generation tasks, we directly compare the LM's prediction with the answer and measure the EM score. When evaluating with CoT Evaluation, smaller LMs including Flan-T5 often do not generate any rationales even with the trigger phrase 'Let's think step by step'. Therefore, we adopt a hard constraint of requiring the LM to generate $r_i^t$ with at least a minimum length of 8 tokens. In classification tasks, we divide into two steps where the LM first generates $r_i^t$, and then verbalizers are applied with a indicator phrase '[ANSWER]' inserted between $r_i^t$ and the possible options. For generation tasks, we extract the output coming after the indicator phrase. Accuracy metric is used for classification tasks while EM metric is used for generation tasks.

Citation

@misc{kim2023cotcollection,
  title={The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning},
  author={Kim et al. (2023)},
  year={2023},
  note={arXiv:2305.14045}
}

• arXiv: 2305.14045