# T5 Eval

> Benchmarks a text-to-text transformer across GLUE, SuperGLUE, CNN/Daily Mail, SQuAD, and WMT, reporting GLUE average, BLEU, ROUGE-2-F, and Exact Match scores.

- Skill: `qhjqhj00/t5-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/t5-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/t5-eval/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML, Model Training & Fine-tuning
- Tags: Benchmark, Bleu, Glue, Nlp, Rouge, Squad, Superglue, T5
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-08-22
- Page: https://skillmd.com/skills/qhjqhj00/t5-eval

---


# t5-eval

> Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — Colin Raffel et al. (arXiv:1910.10683, 2019)

## What this evaluates

Evaluates a unified text-to-text transformer's ability to generalize across diverse NLP tasks including language understanding, summarization, question answering, and machine translation. The protocol tests the efficacy of pre-training objectives, data scaling, and consistent text-to-text fine-tuning pipelines.

## Datasets

- **GLUE** — total ?; splits: val (-1)
- **SuperGLUE** — total ?; splits: val (-1)
- **CNN/Daily Mail** — total ?; splits: val (-1)
- **SQuAD** — total ?; splits: val (-1)
- **WMT** — total ?; splits: val (-1)

## Metrics

- `GLUE average score` **(primary)** — range: [0, 100]
  - Average of scores across all GLUE subtasks, as stipulated by the official benchmark.
- `BLEU` — range: [0, 100]
  - BLEU score computed using SacreBLEU v1.3.0 with 'exp' smoothing and 'intl' tokenization.
- `ROUGE-2-F` — range: [0, 1]
  - ROUGE-2 F1 score, reported as a proxy for ROUGE-1-F and ROUGE-L-F due to high correlation on CNN/Daily Mail.
- `Exact Match` — range: [0, 1]
  - Exact match score for SQuAD, reported as a proxy for F1 due to high correlation.

## Input / output format

**Input**: Text string formatted as a task-specific prefix followed by the input content (e.g., 'translate English to German: ...'). All tasks are unified into a text-to-text format.

**Output**: Text string generated via greedy decoding, representing the target answer, label, or translation.

## Scoring recipe

```python
def compute_metrics(predictions, golds, metric):
    if metric == 'glue_avg':
        return mean([compute_subtask_score(p, g) for p, g in zip(predictions, golds)])
    elif metric == 'bleu':
        return sacrebleu.corpus_bleu(predictions, [golds], smooth_method='exp', tokenize='intl')
    elif metric == 'rouge2f':
        return rouge_score(golds, predictions, rouge_types=['rouge2']).rouge2.fmeasure
    elif metric == 'exact_match':
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
```

## Common pitfalls

- Results are reported on validation sets to avoid test set leakage, not on held-out test sets.
- High inter-run variance on low-resource tasks (CoLA, CB, COPA) can disproportionately skew GLUE and SuperGLUE averages.
- Proxy metrics (ROUGE-2-F, Exact Match) are used instead of full metric suites due to high correlation, which may mask task-specific performance differences.

## Evidence (verbatim from paper)

> For GLUE and SuperGLUE, we report the average score across all subtasks (as stipulated by the official benchmarks) under the headings "GLUE" and "SGLUE". For all translation tasks, we report the BLEU score (Papineni et al., 2002) as provided by SacreBLEU v1.3.0 (Post, 2018) with "exp" smoothing and "intl" tokenization. ... For CNN/Daily Mail, we find the performance of models on the ROUGE-1-F, ROUGE-2-F, and ROUGE-L-F metrics (Lin, 2004) to be highly correlated so we report the ROUGE-2-F score alone under the heading "CNNDM". Similarly, for SQuAD we find the performance of the "exact match" and "F1" scores to be highly correlated so we report the "exact match" score alone.

## Citation

```bibtex
@misc{raffel2019t5,
  title={Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
  author={Colin Raffel et al.},
  year={2019},
  note={arXiv:1910.10683}
}
```

- arXiv: 1910.10683