Few Shot Nlp Eval

few-shot-nlp-eval

Transcending Scaling Laws with 0.1% Extra Compute — Yi Tay et al. (2022) (arXiv:2210.11399, 2022)

What this evaluates

Evaluates the zero-shot and few-shot capabilities of large language models across a diverse suite of NLP, reasoning, and commonsense benchmarks. It measures how efficiently a model scales with compute and whether additional training objectives unlock emergent reasoning abilities.

Datasets

• GPT-3 suite — total ?; splits: test (-1)
• BigBench Emergent Suite — total ?; splits: test (-1)
• Commonsense QA benchmarks — total ?; splits: test (-1)
• Closed-book QA benchmarks — total ?; splits: test (-1)

Metrics

• average score (primary) — range: percent
  • Un-normalized arithmetic mean of per-task accuracy scores across all evaluated tasks.
• exact-match accuracy — range: percent
  • Fraction of predictions that exactly match the ground-truth answer or label.

Input / output format

Input: Zero-shot or few-shot natural language prompts (e.g., 5-shot for BigBench tasks, 0-shot or few-shot for NLP benchmarks), often with task-specific templates or chain-of-thought instructions.

Output: Natural language text or discrete answer tokens corresponding to the task's expected output format.

Scoring recipe

def compute_average_score(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        scores.append(1.0 if pred.strip() == gold.strip() else 0.0)
    return sum(scores) / len(scores) * 100

Common pitfalls

• Averages are computed over un-normalized task scores, which can be skewed by tasks with different score ranges or difficulty levels.
• Chain-of-thought prompting is used for reasoning tasks but explicitly avoided for the BigBench Emergent Suite, making direct comparisons with CoT-tuned models invalid.
• Savings rates and scaling curves are reported in FLOPs rather than wall-clock time or tokens processed, complicating real-world efficiency comparisons.

Evidence (verbatim from paper)

For evaluation, we use the average score of NLU and NLG tasks from the GPT-3 suite*(Brown et al., 2020)*. In total we use 26 tasks (e.g., TriviaQA, NaturalQuestions, SuperGLUE, PIQA, OpenbookQA, ANLI etc). Details and exact scores for Figure 2 can be found in the Appendix.

Citation

@misc{tay2022transcending,
  title={Transcending Scaling Laws with 0.1% Extra Compute},
  author={Yi Tay et al. (2022)},
  year={2022},
  note={arXiv:2210.11399}
}

• arXiv: 2210.11399