Glue Benchmark Eval

glue-benchmark-eval

Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity — Svirsky et al. (2026) (arXiv:2602.09169, 2026)

What this evaluates

Evaluates the ability of efficient fine-tuning and structured sparsity methods to maintain performance across a diverse suite of natural language understanding tasks. It probes task-specific classification accuracy and correlation metrics under both full-data and limited-data regimes, as well as pre-training perplexity on a large-scale corpus.

Datasets

• GLUE benchmark — total ?; splits: test (-1); HF glue

Metrics

• GLUE average score (primary) — range: [0, 1]
  • Arithmetic mean of task-specific metrics: accuracy for most tasks, Matthew's correlation for CoLA, and Pearson correlation for STSB. MNLI uses overall (matched and mismatched) accuracy.
• accuracy — range: [0, 1]
  • Proportion of correctly predicted class labels out of total samples.
• Matthew's correlation — range: [-1, 1]
  • Correlation coefficient measuring the quality of binary and multi-class classifications, accounting for true/false positives and negatives.
• Perplexity — range: other
  • Exponential of the average negative log-likelihood of the validation tokens on the C4 dataset.

Input / output format

Input: Tokenized text sequences or sentence pairs for classification tasks; raw text tokens for C4 pre-training.

Output: Predicted class labels for classification tasks; probability distributions or log-likelihoods for perplexity computation.

Scoring recipe

def compute_glue_score(predictions, golds):
    scores = []
    for task in ['CoLA', 'STSB', 'MRPC', 'RTE', 'SST2', 'MNLI', 'QNLI', 'QQP']:
        if task == 'CoLA':
            scores.append(matthews_corrcoef(golds[task], predictions[task]))
        elif task == 'STSB':
            scores.append(pearsonr(golds[task], predictions[task]))
        else:
            scores.append(accuracy_score(golds[task], predictions[task]))
    return sum(scores) / len(scores)

Common pitfalls

• MNLI evaluation must combine matched and mismatched test sets into a single overall accuracy score rather than reporting them separately.
• Task-specific metrics differ (accuracy vs. correlation), so the GLUE average is a simple arithmetic mean of these heterogeneous scores, not a weighted average.
• Limited-data experiments use the first 10,000 samples from large datasets rather than standard train/val/test splits.

Evidence (verbatim from paper)

We report the overall (matched and mismatched) accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STSB, and accuracy for other tasks.

Citation

@misc{svirsky2026finegates,
  title={Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity},
  author={Svirsky et al. (2026)},
  year={2026},
  note={arXiv:2602.09169}
}

• arXiv: 2602.09169