Glue Eval

glue-eval

Confidence Regularized Masked Language Modeling using Text Length — Ji et al. (2025) (arXiv:2504.06037, 2025)

What this evaluates

Evaluates the transfer learning capability of pre-trained language models across a diverse suite of natural language understanding tasks, including sentence classification, semantic textual similarity, and natural language inference.

Datasets

• GLUE — total ?; splits: test (-1); HF glue

Metrics

• GLUE Average (primary) — range: percent
  • Arithmetic mean of task-specific scores: Matthew's correlation for CoLA, Pearson correlation for STS-b, and accuracy for all other tasks. WNLI is explicitly excluded from the average.

Input / output format

Input: Tokenized text sequences up to 512 tokens, grouped by length during pre-training; standard task-specific prompts during fine-tuning.

Output: Class labels for classification tasks; confidence scores for each class used to compute calibration metrics.

Scoring recipe

def compute_glue_avg(preds, golds, tasks):
    scores = []
    for task in tasks:
        if task == 'CoLA': scores.append(matthews_corrcoef(golds[task], preds[task]))
        elif task == 'STS-b': scores.append(pearsonr(golds[task], preds[task]))
        else: scores.append(accuracy_score(golds[task], preds[task]))
    return sum(scores) / len(scores)

Common pitfalls

• Excluding WNLI from the GLUE average as specified in the paper.
• Reporting task-specific metrics (Matthew's corr, Pearson corr, accuracy) instead of the aggregated average.
• Not averaging results over 7 different random seeds as reported in the tables.

Evidence (verbatim from paper)

We evaluated methods on the GLUE benchmark (Wang et al., 2019) and SQuAD 1.1/2.0 datasets (Rajpurkar et al., 2016, 2018). Following Devlin et al. (2019), we excluded WNLI from tasks of GLUE benchmark. We reported Matthew's correlation score for CoLA, Pearson correlations for STS-b, F1 score for SQuAD 1.1/2.0, and accuracy scores for the other tasks.

Citation

@misc{ji2025confidenceregularized,
  title={Confidence Regularized Masked Language Modeling using Text Length},
  author={Ji et al. (2025)},
  year={2025},
  note={arXiv:2504.06037}
}

• arXiv: 2504.06037