Natural Instructions Eval

natural-instructions-eval

Cross-Task Generalization via Natural Language Crowdsourcing Instructions — Mishra et al. (2021) (arXiv:2104.08773, 2021)

What this evaluates

Evaluates a model's ability to generalize to unseen NLP tasks by leveraging crowdsourced natural language instructions alongside training data. It measures how well instruction-based learning transfers across different task categories, datasets, and individual tasks compared to data-only training.

Datasets

• Natural Instructions — total 193000; splits: train (-1), test (-1); repo https://github.com/allenai/natural-instructions-v1

Metrics

• ROUGE-L (primary) — range: [0, 1]
  • Automated evaluation metric for text generation that computes the longest common subsequence between predicted and reference outputs. The paper treats all 61 tasks as text generation problems and uses this metric to score generated outputs.

Input / output format

Input: A natural language instruction (potentially containing definition, prompt, positive examples, and negative examples) concatenated with the task-specific input instance.

Output: A generated text sequence corresponding to the input instance (e.g., answer, question, or classification label). For GPT3, outputs are limited to a maximum of 16 tokens with a stop condition of 2 newline tokens.

Scoring recipe

def compute_rouge_l(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        # Compute ROUGE-L F1 score between reference and prediction
        score = rouge_l_score(ref, pred)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Negative examples in instructions often degrade model performance, contrary to human intuition and prior assumptions about their utility.
• ROUGE-L is applied uniformly across all 61 tasks (including classification and verification), which may not capture task-specific correctness accurately.
• Task splits are defined at the task level (not instance level), so improper shuffling can cause data leakage between train and test sets.

Evidence (verbatim from paper)

We treat all of our tasks as text generation problems and evaluate them with automated evaluation metrics for text generation. In particular, we use ROUGE-LLin (2004) to automatically evaluate the generated outputs.777Our experiments show that other metrics, e.g. BLEURTSellam et al. (2020) are also correlated with ROUGE-L, which has also been used in generative QA tasks.

Citation

@misc{mishra2021naturalinstructions,
  title={Cross-Task Generalization via Natural Language Crowdsourcing Instructions},
  author={Mishra et al. (2021)},
  year={2021},
  note={arXiv:2104.08773}
}

• arXiv: 2104.08773