Glue Squad Eval

glue-squad-eval

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models — Zhang et al. (2024) (arXiv:2407.02775, 2024)

What this evaluates

Evaluates the ability of pre-trained language models to perform a diverse set of natural language understanding tasks, including sentence classification, paraphrase detection, semantic similarity, natural language inference, and extractive question answering.

Datasets

• GLUE — total ?; splits: test (-1); HF glue; repo https://gluebenchmark.com
• SQuAD 1.1 — total ?; splits: dev (-1); HF squad
• SQuAD 2.0 — total ?; splits: dev (-1); HF squad_v2

Metrics

• Task-specific metrics (Accuracy, F1, Spearman, Matthews) (primary) — range: [0, 1] | percent
  • Accuracy for SST-2, QNLI, RTE, MNLI-m, MNLI-mm; F1 for MRPC, QQP, SQuAD 1.1/2.0; Spearman correlation for STS-B; Matthew’s correlation for CoLA. The paper also reports an unweighted average across all GLUE tasks.

Input / output format

Input: Standard NLP task formats: sentence pairs or single sentences for GLUE tasks; passage-question pairs for SQuAD tasks.

Output: Class labels for GLUE classification tasks; predicted text spans for SQuAD extractive QA.

Scoring recipe

def score(predictions, golds, task):
    if task in ['SST-2', 'QNLI', 'RTE', 'MNLI-m', 'MNLI-mm']:
        return accuracy(golds, predictions)
    elif task in ['MRPC', 'QQP', 'SQuAD 1.1', 'SQuAD 2.0']:
        return f1(golds, predictions)
    elif task == 'STS-B':
        return spearman_corr(golds, predictions)
    elif task == 'CoLA':
        return matthews_corr(golds, predictions)

Common pitfalls

• GLUE test set scores are evaluated on the official test set, which is not publicly released; results may not be reproducible without the official test data or submission to the GLUE leaderboard.
• Averaging heterogeneous metrics (Accuracy, F1, Spearman, Matthews) across GLUE tasks to report a single 'average performance' is mathematically inconsistent and can misrepresent model capability.
• SQuAD 2.0 evaluation requires handling unanswerable questions with a specific thresholding strategy; the paper only mentions 'F1 metric' without detailing the unanswerable prediction handling.

Evidence (verbatim from paper)

The GLUE tasks are evaluated on GLUE test sets and the extractive question answering tasks are evaluated on dev sets. Here, different evaluation indices are adopted with regarding to the tasks: F1 metric for MRPC and QQP, Spearman correlation for STS-B, Matthew’s correlation for CoLA, and Accuracy for the other tasks. The comparative results evaluated on the dev sets of SQuAD 1.1 and SQuAD 2.0 are presented in Table [2], with F1 metric for evaluation.

Citation

@misc{zhang2024mlkdbert,
  title={MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2407.02775}
}

• arXiv: 2407.02775