Palm Fewshot Nlp Eval

palm-fewshot-nlp-eval

PaLM: Scaling Language Modeling with Pathways — Chowdhery et al. (2022) (arXiv:2204.02311, 2022)

What this evaluates

Evaluates the few-shot and fine-tuned capabilities of large autoregressive language models across a wide range of English NLP benchmarks, including question answering, reading comprehension, common sense reasoning, and natural language inference. It also assesses performance on a large collection of collaborative reasoning and language tasks to probe multi-step reasoning and general language understanding.

Datasets

• English NLP Benchmarks (29 tasks) — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• BIG-bench (textual) — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly predicted options or labels for multiple-choice and classification tasks.
• exact-match accuracy — range: [0, 1]
  • Percentage of generated answers that exactly match the gold label string.
• F1 score — range: [0, 1]
  • Harmonic mean of token-level precision and recall between the generated answer and the gold answer.
• preferred metric — range: [0, 100]
  • Task-specific metric normalized to [0, 100] by setting the maximum possible score to 100 and random chance to 0.

Input / output format

Input: Autoregressive text prompts containing task instructions and k-shot exemplars (0, 1, or few-shot with k ranging from 2 to 100 depending on the task). For fine-tuning, a task-proportionate mixture of SuperGLUE tasks is used.

Output: Autoregressive text completion predicting the next token(s) until a stop condition or maximum length. For multiple-choice tasks, the model generates the answer option text.

Scoring recipe

def score(predictions, golds, metric_type):
    if metric_type == 'exact-match':
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    elif metric_type == 'f1':
        return compute_f1(predictions, golds)
    elif metric_type == 'accuracy':
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    elif metric_type == 'normalized':
        raw = compute_task_metric(predictions, golds)
        return normalize_to_0_100(raw)

Common pitfalls

• Using different few-shot shot counts (k) across tasks without reporting the exact k used, making cross-task comparison difficult.
• Comparing results from models that use fine-tuning or multi-task adaptation against purely few-shot pretrained models, as the paper explicitly excludes them from primary comparisons.
• Ignoring the normalization convention for BIG-bench tasks, where scores are scaled to [0, 100] with random chance at 0, which can produce negative values if performance is below random.

Evidence (verbatim from paper)

For each task, the results of its preferred metric are used. The results are normalized by setting the maximum score to 100 and the random chance score to 0 for multiple-choice tasks, so that they are negative valued if the model performs worse than random chance.

Citation

@misc{chowdhery2022palm,
  title={PaLM: Scaling Language Modeling with Pathways},
  author={Chowdhery et al. (2022)},
  year={2022},
  note={arXiv:2204.02311}
}

• arXiv: 2204.02311