Gpt3 Few Shot Eval

gpt3-few-shot-eval

Language Models are Few-Shot Learners — Tom B. Brown et al. (2020) (arXiv:2005.14165, 2020)

What this evaluates

Evaluates the few-shot, one-shot, and zero-shot learning capabilities of large autoregressive language models across diverse NLP tasks including language modeling, cloze completion, question answering, translation, and commonsense reasoning.

Datasets

• Penn Tree Bank (PTB) — total ?; splits: test (-1)
• LAMBADA — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• StoryCloze 2016 — total ?; splits: test (-1)
• Natural Questions — total ?; splits: test (-1)
• WebQuestions — total ?; splits: test (-1)
• TriviaQA — total ?; splits: test (-1)
• WMT14/WMT16 Translation — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of predictions that exactly match the gold answer string.
• perplexity — range: [0, ∞)
  • Exponential of the average cross-entropy loss over the test set.
• BLEU — range: [0, 100]
  • Multi-bleu score using XLM tokenization, as measured by multi-bleu.perl.

Input / output format

Input: Text prompts containing task instructions and K demonstrations (examples), followed by the query to be completed. For zero-shot, only the instruction or raw text is provided.

Output: Autoregressive text completion. For cloze/QA tasks, a single word or short answer. For translation, the translated sentence.

Scoring recipe

def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    ppl = math.exp(sum(-math.log(p) for p in model_probs) / len(golds))
    bleu = sacrebleu.corpus_bleu(predictions, [golds])
    return {'accuracy': acc, 'perplexity': ppl, 'bleu': bleu}

Common pitfalls

• Data contamination in training set affects some benchmarks (e.g., LAMBADA, TriviaQA).
• Zero-shot vs few-shot formatting differences significantly impact performance (e.g., LAMBADA requires fill-in-the-blank framing for few-shot).
• PTB only evaluated zero-shot due to lack of clear few-shot split.

Evidence (verbatim from paper)

We evaluate all tasks in the few-shot, one-shot, and zero-shot settings. ... We calculate zero-shot perplexity on the Penn Tree Bank (PTB) dataset ... GPT-3 achieves 86.4% accuracy in the few-shot setting ... We report BLEU scores on the WMT'14 Fr↔En WMT'16 De↔En, and WMT'16 Ro↔En datasets as measured by multi-bleu.perl with XLM's tokenization in order to compare most closely with prior unsupervised NMT work.

Citation

@misc{brown2020gpt3,
  title={Language Models are Few-Shot Learners},
  author={Tom B. Brown et al. (2020)},
  year={2020},
  note={arXiv:2005.14165}
}

• arXiv: 2005.14165