# Glue Eval

> Evaluates the transfer learning capability of pre-trained language models across a diverse suite of natural language understanding tasks, including sentence classification, semantic textual similarity, and natural language inference. Use when the user wants to benchmark on GLUE, or asks about evaluating this task. Reports GLUE Average.

- Skill: `qhjqhj00/glue-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/glue-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/glue-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/glue-eval

---


# glue-eval

> Confidence Regularized Masked Language Modeling using Text Length — Ji et al. (2025) (arXiv:2504.06037, 2025)

## What this evaluates

Evaluates the transfer learning capability of pre-trained language models across a diverse suite of natural language understanding tasks, including sentence classification, semantic textual similarity, and natural language inference.

## Datasets

- **GLUE** — total ?; splits: test (-1); HF `glue`

## Metrics

- `GLUE Average` **(primary)** — range: percent
  - Arithmetic mean of task-specific scores: Matthew's correlation for CoLA, Pearson correlation for STS-b, and accuracy for all other tasks. WNLI is explicitly excluded from the average.

## Input / output format

**Input**: Tokenized text sequences up to 512 tokens, grouped by length during pre-training; standard task-specific prompts during fine-tuning.

**Output**: Class labels for classification tasks; confidence scores for each class used to compute calibration metrics.

## Scoring recipe

```python
def compute_glue_avg(preds, golds, tasks):
    scores = []
    for task in tasks:
        if task == 'CoLA': scores.append(matthews_corrcoef(golds[task], preds[task]))
        elif task == 'STS-b': scores.append(pearsonr(golds[task], preds[task]))
        else: scores.append(accuracy_score(golds[task], preds[task]))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Excluding WNLI from the GLUE average as specified in the paper.
- Reporting task-specific metrics (Matthew's corr, Pearson corr, accuracy) instead of the aggregated average.
- Not averaging results over 7 different random seeds as reported in the tables.

## Evidence (verbatim from paper)

> We evaluated methods on the GLUE benchmark (Wang et al., 2019) and SQuAD 1.1/2.0 datasets (Rajpurkar et al., 2016, 2018). Following Devlin et al. (2019), we excluded WNLI from tasks of GLUE benchmark. We reported Matthew's correlation score for CoLA, Pearson correlations for STS-b, F1 score for SQuAD 1.1/2.0, and accuracy scores for the other tasks.

## Citation

```bibtex
@misc{ji2025confidenceregularized,
  title={Confidence Regularized Masked Language Modeling using Text Length},
  author={Ji et al. (2025)},
  year={2025},
  note={arXiv:2504.06037}
}
```

- arXiv: 2504.06037