Opt Iml Bench Eval

opt-iml-bench-eval

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — Iyer et al. (2022) (arXiv:2212.12017, 2022)

What this evaluates

Evaluates instruction-tuned language models on generalization across 1,991 NLP tasks spanning 100+ categories. It probes zero-shot and few-shot (5-shot) performance on held-out categories, unseen tasks within seen categories, and fully supervised tasks, measuring both generation quality and classification accuracy.

Datasets

• OPT-IML Bench — total 1991; splits: test (-1)

Metrics

• Rouge-L (primary) — range: [0, 100] | percent
  • Longest common subsequence overlap between the model's generated text and the reference answer, reported as a percentage.
• Accuracy — range: [0, 100] | percent
  • Percentage of predictions that exactly match the gold label or answer.

Input / output format

Input: Instruction prompt with 0 or 5 demonstration examples (task-specific format).

Output: Free-form text generation or classification label.

Scoring recipe

def compute_metrics(predictions, golds, metric_type):
    if metric_type == 'accuracy':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds) * 100
    elif metric_type == 'rouge-l':
        scores = [rouge_l_score(p, g) * 100 for p, g in zip(predictions, golds)]
        return sum(scores) / len(scores)
    return None

Common pitfalls

• Results are reported in 0-shot/5-shot format; summarization tasks only report 0-shot performance.
• Averages across task categories mix Rouge-L and accuracy scores, which are on different scales and should not be directly compared without normalization.
• Some evaluation tables use 32-shot instead of 5-shot (e.g., Table 16), so shot counts must be verified per table.

Evidence (verbatim from paper)

Most tasks are generation tasks, for which we report Rouge-L. We report accuracy for MMLU. Some tasks in the Cause Effect Cluster also use accuracy, which is averaged with Rouge-L for presentation purposes.

Citation

@misc{iyer2022optiml,
  title={OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization},
  author={Iyer et al. (2022)},
  year={2022},
  note={arXiv:2212.12017}
}

• arXiv: 2212.12017