Fewshot Ft Vs Icl Eval

fewshot-ft-vs-icl-eval

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation — Mosbach et al. (2023) (arXiv:2305.16938, 2023)

What this evaluates

Evaluates the in-domain and out-of-domain generalization capabilities of large language models adapted via few-shot fine-tuning versus in-context learning across standard natural language inference and paraphrase detection benchmarks.

Datasets

• MNLI — total ?; splits: train (-1), test (-1)
• RTE — total ?; splits: train (-1), test (-1)
• QQP — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the fraction of correctly predicted labels out of the total number of test instances.

Input / output format

Input: Natural language inference and paraphrase detection sentences formatted as few-shot prompts with 16 demonstrations (ICL) or as fine-tuning examples (FT).

Output: Classification labels (e.g., entailment/contradiction/neutral for MNLI; entailment/not_entailment for RTE; same/different for QQP).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Prior studies often compared ICL and FT using mismatched model sizes, leading to biased conclusions about OOD robustness.
• Model selection strategy (e.g., picking checkpoints based on in-domain vs. OOD performance) drastically changes reported OOD generalization results.
• Both methods are highly sensitive to prompt design and training instability, requiring multiple random seeds for reliable evaluation.

Evidence (verbatim from paper)

We present the results for in-domain and OOD model performance in Figure 2, comparing both ICL and FT. We perform task adaptation using 16 examples for both strategies. ... On MNLI the largest model (30B) obtains an average performance of 71.4% and a maximum performance of 74.9%. ... — in the x- and y-axis indicates majority class accuracy.

Citation

@misc{mosbach2023fewshot,
  title={Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation},
  author={Mosbach et al. (2023)},
  year={2023},
  note={arXiv:2305.16938}
}

• arXiv: 2305.16938