Gem Cot Mixed Task Eval

gem-cot-mixed-task-eval

Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models — Anni Zou et al. (2023) (arXiv:2310.06692, 2023)

What this evaluates

Evaluates the ability of LLMs to perform zero-shot and few-shot reasoning across a heterogeneous mix of unseen and known task types without manual task-specific prompting. It probes dynamic demonstration routing, clustering-based generalization, and streaming adaptation in mixed-task scenarios.

Datasets

• AQUA-RAT — total ?; splits: test (-1)
• MultiArith — total ?; splits: test (-1)
• AddSub — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• SingleEq — total ?; splits: test (-1)
• SVAMP — total ?; splits: test (-1)
• Last Letter Concatenation — total ?; splits: test (-1)
• Coin Flip — total ?; splits: test (-1)
• StrategyQA — total ?; splits: test (-1)
• CSQA — total ?; splits: test (-1)
• BIG-Bench Hard (BBH) — total 2000; splits: test (2000)

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly predicted answers out of the total number of test instances. Calculated as (number of exact matches between model output and gold answer) / (total predictions) * 100.

Input / output format

Input: A single natural language question or problem statement.

Output: A chain-of-thought reasoning trace followed by a final answer.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
    return (correct / len(golds)) * 100

Common pitfalls

• The BBH evaluation uses a shuffled, randomly sampled subset of 2000 questions to simulate mixed-task scenarios, rather than the official full test set.
• The method dynamically updates its demonstration pool via density-based clustering during inference, so static few-shot baselines cannot be directly compared without accounting for this continuous learning aspect.

Evidence (verbatim from paper)

Table 3: Accuracy (%) on ten reasoning datasets. The backbone model is GPT-3.5-Turbo. Results in bold and underline are the best and second-best performances, respectively. ... For the BBH (Suzgun et al., 2022) tasks, we shuffle all the data and randomly sample 2000 questions to imitate the realistic mixed-task scenarios.

Citation

@misc{zou2023gemcot,
  title={Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models},
  author={Anni Zou et al. (2023)},
  year={2023},
  note={arXiv:2310.06692}
}

• arXiv: 2310.06692