# Glue Benchmark Eval

> Evaluates the ability of efficient fine-tuning and structured sparsity methods to maintain performance across a diverse suite of natural language understanding tasks. It probes task-specific classification accuracy and correlation metrics under both full-data and limited-data regimes, as well as pre-training perplexity on a large-scale corpus. Use when the user wants to benchmark on GLUE benchmark, or asks about evaluating this task. Reports GLUE average score.

- Skill: `qhjqhj00/glue-benchmark-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/glue-benchmark-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/glue-benchmark-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/glue-benchmark-eval

---


# glue-benchmark-eval

> Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity — Svirsky et al. (2026) (arXiv:2602.09169, 2026)

## What this evaluates

Evaluates the ability of efficient fine-tuning and structured sparsity methods to maintain performance across a diverse suite of natural language understanding tasks. It probes task-specific classification accuracy and correlation metrics under both full-data and limited-data regimes, as well as pre-training perplexity on a large-scale corpus.

## Datasets

- **GLUE benchmark** — total ?; splits: test (-1); HF `glue`

## Metrics

- `GLUE average score` **(primary)** — range: [0, 1]
  - Arithmetic mean of task-specific metrics: accuracy for most tasks, Matthew's correlation for CoLA, and Pearson correlation for STSB. MNLI uses overall (matched and mismatched) accuracy.
- `accuracy` — range: [0, 1]
  - Proportion of correctly predicted class labels out of total samples.
- `Matthew's correlation` — range: [-1, 1]
  - Correlation coefficient measuring the quality of binary and multi-class classifications, accounting for true/false positives and negatives.
- `Perplexity` — range: other
  - Exponential of the average negative log-likelihood of the validation tokens on the C4 dataset.

## Input / output format

**Input**: Tokenized text sequences or sentence pairs for classification tasks; raw text tokens for C4 pre-training.

**Output**: Predicted class labels for classification tasks; probability distributions or log-likelihoods for perplexity computation.

## Scoring recipe

```python
def compute_glue_score(predictions, golds):
    scores = []
    for task in ['CoLA', 'STSB', 'MRPC', 'RTE', 'SST2', 'MNLI', 'QNLI', 'QQP']:
        if task == 'CoLA':
            scores.append(matthews_corrcoef(golds[task], predictions[task]))
        elif task == 'STSB':
            scores.append(pearsonr(golds[task], predictions[task]))
        else:
            scores.append(accuracy_score(golds[task], predictions[task]))
    return sum(scores) / len(scores)
```

## Common pitfalls

- MNLI evaluation must combine matched and mismatched test sets into a single overall accuracy score rather than reporting them separately.
- Task-specific metrics differ (accuracy vs. correlation), so the GLUE average is a simple arithmetic mean of these heterogeneous scores, not a weighted average.
- Limited-data experiments use the first 10,000 samples from large datasets rather than standard train/val/test splits.

## Evidence (verbatim from paper)

> We report the overall (matched and mismatched) accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STSB, and accuracy for other tasks.

## Citation

```bibtex
@misc{svirsky2026finegates,
  title={Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity},
  author={Svirsky et al. (2026)},
  year={2026},
  note={arXiv:2602.09169}
}
```

- arXiv: 2602.09169