qhjqhj00 glue-sni-eval
The MiniPile Challenge for Data-Efficient Language Models — Kaddour et al. (2023) (arXiv:2304.08442, 2023)
What this evaluates
Evaluates the downstream generalization and data efficiency of pre-trained language models by fine-tuning them on standard natural language understanding and instruction-following benchmarks.
Datasets
- GLUE — total ?; splits: train (-1), dev (-1), test (-1)
- SuperNatural-Instructions (SNI) — total ?; splits: train (-1), test (-1)
Metrics
GLUE(primary) — range: percent- Macro-averaged performance across all GLUE sub-tasks, using accuracy, F1, or correlation as appropriate per task, then averaged. Reported as a percentage or normalized score.
SNI— range: percent- Exact-match or token-level accuracy across instruction-following tasks, aggregated per task and averaged.
Input / output format
Input: Task-specific prompts, instruction-response pairs, or sentence pairs for fine-tuning and evaluation.
Output: Predicted labels, text completions, or numerical answers corresponding to each downstream task.
Scoring recipe
def compute_metrics(predictions, golds, task_names):
scores = []
for preds, golds, task in zip(predictions, golds, task_names):
if task in ['cola', 'sst-2', 'mrpc', 'qqp', 'mnli', 'qnli', 'rte', 'wnli']:
scores.append(accuracy_score(golds, preds))
elif task == 'stsb':
scores.append(pearsonr(golds, preds)[0])
else:
scores.append(f1_score(golds, preds, average='macro'))
return sum(scores) / len(scores) * 100
Common pitfalls
- GLUE contains diverse sub-tasks with different evaluation metrics (accuracy, F1, correlation); results must be averaged correctly per official GLUE guidelines.
- SNI covers highly varied instruction formats; exact-match vs. token-level scoring can vary significantly across tasks and must be applied consistently.
- Comparisons against baselines require identical fine-tuning hyperparameters and data splits to isolate the effect of pre-training corpus quality.
Evidence (verbatim from paper)
We evaluate our pre-trained models on the General Language Understanding Evaluation (GLUE) [55] and SuperNatural-Instructions (SNI) [57] benchmarks. ... Tables 3 and 4 show the results compared against the publicly available checkpoints trained on 2.6x / 745x the amount of data... We observe minor reductions in final downstream performance and conjecture that MiniPile is a well-suited pre-training corpus for common downstream benchmarks.
Citation
@misc{kaddour2023minipile,
title={The MiniPile Challenge for Data-Efficient Language Models},
author={Kaddour et al. (2023)},
year={2023},
note={arXiv:2304.08442}
}
- arXiv: 2304.08442