Mcl Eval

mcl-eval

Pre-training Language Model as a Multi-perspective Course Learner — Chen et al. (2023) (arXiv:2305.03981, 2023)

What this evaluates

Evaluates the downstream performance and sample efficiency of the MCL pre-trained language model on general language understanding and reading comprehension benchmarks. It measures how well the model captures multi-perspective semantics and self-corrects during pre-training when fine-tuned on standard NLP tasks.

Datasets

• GLUE benchmark — total ?; splits: train (-1), dev (-1), test (-1)
• SQuAD 2.0 — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• GLUE Average (primary) — range: percent
  • Arithmetic mean of task-specific scores across 8 GLUE tasks. Uses accuracy for most tasks, Matthews correlation for CoLA, and Spearman correlation for STS-B.
• ExactMatch (EM) — range: percent
  • Fraction of questions where the predicted answer span exactly matches one of the gold answer spans.
• F1 — range: percent
  • Token-level F1 score between predicted and gold answer spans, averaging precision and recall.

Input / output format

Input: Tokenized text sequences up to 512 tokens. For downstream tasks: sentence pairs or question-passage pairs.

Output: Class labels for GLUE tasks; character or token spans for SQuAD 2.0 answers.

Scoring recipe

def compute_glue_avg(preds_dict, golds_dict):
    scores = []
    for task in preds_dict:
        if task == 'CoLA': scores.append(matthews_corrcoef(golds_dict[task], preds_dict[task]))
        elif task == 'STS-B': scores.append(spearmanr(golds_dict[task], preds_dict[task]).correlation)
        else: scores.append(accuracy_score(golds_dict[task], preds_dict[task]))
    return mean(scores)

def compute_squad(preds, golds):
    em = sum(1 for p, g in zip(preds, golds) if p == g) / len(preds)
    f1 = token_f1(preds, golds)
    return em, f1

Common pitfalls

• Report arithmetic mean instead of median over 5 random seeds for GLUE results.
• Apply accuracy metric to CoLA or STS-B instead of Matthews correlation or Spearman correlation.
• Mix up MNLI matched vs. mismatched evaluation sets.

Evidence (verbatim from paper)

As for the evaluation metrics of GLUE tasks, we adopt Spearman correlation for STS, Matthews correlation for CoLA, and accuracy for the other. For SQuAD 2.0, in which some questions are unanswerable by the passage, the standard evaluation metrics of ExactMatch (EM) and F1 scores are adopted. We conducted a hyperparameter search for all downstream tasks, and report the average scores among 5 random seeds.

Citation

@misc{chen2023mcl,
  title={Pre-training Language Model as a Multi-perspective Course Learner},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2305.03981}
}

• arXiv: 2305.03981