Flan 2022 Eval

flan-2022-eval

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning — Longpre et al. (2023) (arXiv:2301.13688, 2023)

What this evaluates

Evaluates the instruction-tuning effectiveness of models trained on the Flan 2022 collection across held-in, chain-of-thought, and held-out benchmarks. It probes zero-shot and few-shot generalization capabilities on reasoning, knowledge, and natural language understanding tasks.

Datasets

• MMLU — total 57; splits: test (57)
• BBH — total 23; splits: test (23)

Metrics

• zero-shot/few-shot accuracy (primary) — range: percent
  • Standard exact-match accuracy computed over the validation/test sets for each benchmark. Reported separately for zero-shot and few-shot prompting settings.

Input / output format

Input: Instruction prompts, optionally augmented with few-shot examples or Chain-of-Thought reasoning traces. For held-out tasks, inputs are provided in zero-shot or few-shot formats.

Output: Model-generated text containing the predicted answer or reasoning trace.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Evaluating on held-in tasks without distinguishing between zero-shot and few-shot settings, as performance varies significantly between the two.
• Assuming task scaling beyond 1800 tasks yields linear improvements without considering task diversity and mixture weighting, which can cause performance saturation.

Evidence (verbatim from paper)

We evaluate on (a) a suite of 8 "Held-In" tasks represented within the 1800+ training task collection (4 question answering and 4 natural language inference validation sets), (b) Chain-of-Thought (CoT) tasks (5 validation sets), and (c) the MMLU (Hendrycks et al., 2020) and BBH (Suzgun et al., 2022) benchmarks as our set of "Held-Out" tasks, as they are not included as part of Flan 2022 finetuning. Metrics are reported in both zero-shot / few-shot settings across Held-In, Chain-of-Thought, and Held-Out (MMLU, BBH) tasks.

Citation

@misc{longpre2023flancollection,
  title={The Flan Collection: Designing Data and Methods for Effective Instruction Tuning},
  author={Longpre et al. (2023)},
  year={2023},
  note={arXiv:2301.13688}
}

• arXiv: 2301.13688