Instruction Pretraining Eval

instruction-pretraining-eval

Instruction Pre-Training: Language Models are Supervised Multitask Learners — Cheng et al. (2024) (arXiv:2406.14491, 2024)

What this evaluates

Evaluates the generalization and domain-adaptive capabilities of language models pre-trained with instruction-augmented corpora. Probes zero/few-shot instruction following, general knowledge, and specialized performance in biomedicine and finance.

Datasets

• MMLU — total ?; splits: test (-1)
• PubMedQA — total ?; splits: test (-1)
• ChemProt — total ?; splits: test (-1)
• RCT — total ?; splits: test (-1)
• MQP — total ?; splits: test (-1)
• UMSLE — total ?; splits: test (-1)
• ConvFinQA — total ?; splits: test (-1)
• Headline — total ?; splits: test (-1)
• FiQA SA — total ?; splits: test (-1)
• FPB — total ?; splits: test (-1)
• NER — total ?; splits: test (-1)

Metrics

• average task score (primary) — range: percent
  • Standard dataset-specific accuracy or F1 scores, averaged across all datasets within a domain (e.g., BioMed or Finance).
• MMLU — range: percent
  • Zero/few-shot accuracy on the MMLU benchmark, tracking performance trends during instruction tuning.

Input / output format

Input: Instruction-response pairs for domain tasks; multiple-choice questions for MMLU.

Output: Predicted answer or generated response per instruction/question.

Scoring recipe

domain_scores = []
for dataset in domain_datasets:
    preds = model.generate(inputs)
    scores.append(compute_metric(preds, golds))
domain_scores.extend(scores)
return sum(domain_scores) / len(domain_scores)

Common pitfalls

• Evaluated datasets were explicitly excluded from the instruction synthesizer's fine-tuning data to ensure generalization testing.
• Finance NER benchmark exhibits considerable variance and may be unreliable for strict performance comparisons.

Evidence (verbatim from paper)

Figure[4] shows the zero/few-shot performance on MMLU during instruction tuning from the pre-trained models. The model pre-trained viaInstruction Pre-Trainingquickly outperforms the model pre-trained viaVanilla Pre-Training, and we observe a stable increasing trend of our model throughout the instruction tuning process. As shown in Table[3], Instruction Pre-Trainingconsistently outperforms Vanilla Pre-Trainingon almost all domain-specific tasks. We report the average task scores within each domain.

Citation

@misc{cheng2024instruction,
  title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2406.14491}
}

• arXiv: 2406.14491