Glue Squad2 Eval

glue-squad2-eval

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining — Meng et al. (2021) (arXiv:2102.08473, 2021)

What this evaluates

Evaluates the generalization and downstream performance of pretrained language models on a suite of natural language understanding tasks (GLUE) and reading comprehension (SQuAD 2.0).

Datasets

• GLUE — total ?; splits: train (-1), dev (-1), test (-1)
• SQuAD 2.0 — total ?; splits: train (-1), validation (-1)

Metrics

• GLUE (primary) — range: [0, 1]
  • Macro-average of task-specific metrics across all GLUE tasks.
• SQuAD 2.0 — range: [0, 1]
  • Maximum of Exact Match and F1 scores between predicted and gold answers.

Input / output format

Input: Text pairs or single text sequences for GLUE tasks; question and context passage for SQuAD 2.0.

Output: Class label or regression value for GLUE; span start and end indices (or text) for SQuAD 2.0.

Scoring recipe

def score(predictions, golds, task):
    if task == 'GLUE':
        return mean([compute_metric(p, g) for p, g in zip(predictions, golds)])
    elif task == 'SQuAD 2.0':
        return max(exact_match(predictions, golds), token_f1(predictions, golds))
# Report median across 5 random seeds

Common pitfalls

• Fine-tuning hyperparameters are tuned per task, introducing variance not attributable to pretraining.
• Results are reported as the median of five random seeds rather than the mean, complicating statistical comparisons.
• Pretraining corpora (Wikipedia, BookCorpus, etc.) may overlap with downstream test sets, potentially inflating scores.

Evidence (verbatim from paper)

Downstream Tasks. We use the tasks included in GLUE [54] and SQuAD 2.0 reading compression [41]. Please refer to Appendix A for more details about GLUE tasks. Standard hyperparameter search in fine-tuning is performed, and the search space can be found in Appendix B. The fine-tuning protocols use the open-source implementation of TUPE [26]. The reported results are the median of five random seeds on GLUE and SQuAD.

Citation

@misc{meng2021cocolm,
  title={COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining},
  author={Meng et al. (2021)},
  year={2021},
  note={arXiv:2102.08473}
}

• arXiv: 2102.08473