Super Naturalinstructions Eval

super-naturalinstructions-eval

Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — Wang et al. (2022) (arXiv:2204.07705, 2022)

What this evaluates

Evaluates instruction-following and cross-task generalization capabilities of language models on a massive, diverse benchmark of 1,616 NLP tasks spanning 76 task types and 55 languages. It measures how well models trained on a mix of tasks perform on unseen tasks when given natural language instructions.

Datasets

• Super-NaturalInstructions — total 1616; splits: test (-1)

Metrics

• human evaluation metric (primary) — range: percent
  • The percentage of instances where crowdworkers rated the model's predicted answer as at least as good as the ground truth output. Ties are allowed. The theoretical upper bound is 100%.

Input / output format

Input: Task definition and two positive examples without the negative examples and explanations.

Output: Model's predicted answer for the given task instance.

Scoring recipe

successes = 0
for instance in dataset:
    if crowdworker_rating(model_pred(instance)) >= crowdworker_rating(gold(instance)):
        successes += 1
return (successes / len(dataset)) * 100

Common pitfalls

• InstructGPT's training data is unavailable, so potential overlap with the evaluation tasks is unknown.
• T0's prompting style differs significantly from the benchmark's instruction style, making direct comparison difficult.
• Automatic metrics are only approximations for language generation tasks, so human evaluation is required for reliable assessment.

Evidence (verbatim from paper)

For language generation tasks, automatic metrics are only an approximation of human judgments; we conduct a human evaluation to confirm the findings so far. Specifically, we ask crowdworkers to indicate if they prefer the predicted answer by the model or the ground truth outputs for each instance with ties being allowed... The resulting human evaluation metric indicates how often model predictions were rated as at least as good as our ground truth labels. The theoretical upper bound of this metric is 100% when the model is rated at least as good as the ground truth for all the instances.

Citation

@misc{wang2022supernaturalinstructions,
  title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2204.07705}
}

• arXiv: 2204.07705