Superni Eval

superni-eval

Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. (2022) (arXiv:2212.10560, 2022)

What this evaluates

Evaluates a language model's ability to follow diverse natural language instructions across various NLP tasks in a zero-shot setting. It measures how well the model generalizes to unseen tasks without in-context examples.

Datasets

• SUPER-NATURALINSTRUCTIONS — total 11900; splits: test (11900)

Metrics

• ROUGE-L (primary) — range: percent
  • Measures the longest common subsequence of words between the predicted and reference outputs, normalized by the reference length. Reported as a percentage score.

Input / output format

Input: Task definition (instruction) provided as a prompt, without in-context demonstration examples.

Output: Model-generated text response to the instruction.

Scoring recipe

scores = []
for instance in dataset:
    pred = model.generate(instance.prompt, temperature=0)
    scores.append(rouge_l_score(pred, instance.reference))
return mean(scores)

Common pitfalls

• Using in-context examples despite the zero-shot protocol specified.
• Using non-deterministic generation (temperature > 0) which can cause high variance in ROUGE-L scores.
• Not filtering out prompt tokens from the generated output before scoring.

Evidence (verbatim from paper)

We use the evaluation set of SUPERNI (Wang et al., 2022), which consists of 119 tasks with 100 instances in each task. In this work, we mainly focus on the zero-shot setup, i.e., the model is prompted with the definition of the tasks only, without in-context demonstration examples. Table 3: Evaluation results on unseen tasks from SUPERNI (§4.3). From the results, we see that ① SELFINSTRUCT can boost GPT3 performance by a large margin (+33.1%) and ② nearly matches the performance of InstructGPT001. The table reports ROUGE-L scores.

Citation

@misc{wang2022selfinstruct,
  title={Self-Instruct: Aligning Language Models with Self-Generated Instructions},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2212.10560}
}

• arXiv: 2212.10560