# Wmt21 News Translation Eval

> Evaluates the translation quality and inference speed of non-autoregressive versus autoregressive machine translation models on English-German news text. It probes the practical trade-offs between decoding latency and translation accuracy under realistic deployment conditions. Use when the user wants to benchmark on WMT21 News Translation, or asks about evaluating this task. Reports BLEU.

- Skill: `qhjqhj00/wmt21-news-translation-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/wmt21-news-translation-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/wmt21-news-translation-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/wmt21-news-translation-eval

---


# wmt21-news-translation-eval

> Non-Autoregressive Machine Translation: It's Not as Fast as it Seems — Helcl et al. (2022) (arXiv:2205.01966, 2022)

## What this evaluates

Evaluates the translation quality and inference speed of non-autoregressive versus autoregressive machine translation models on English-German news text. It probes the practical trade-offs between decoding latency and translation accuracy under realistic deployment conditions.

## Datasets

- **WMT21 News Translation** — total ?; splits: train (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram overlap metric between predicted and reference translations, typically computed with sentence-level averaging and smoothing.
- `Latency` — range: ms or words/sec
  - Inference time measured in milliseconds per sentence or words per second, evaluated under varying batch sizes and hardware conditions (CPU vs GPU).

## Input / output format

**Input**: English source sentence (string)

**Output**: German target sentence (string)

## Scoring recipe

```python
def compute_bleu(predictions, references):
    return nltk.translate.bleu_score.corpus_bleu(references, predictions) * 100

def compute_latency(model, inputs, batch_size=1, device='cpu'):
    times = []
    for batch in get_batches(inputs, batch_size):
        start = time.time()
        _ = model.generate(batch, device=device)
        times.append((time.time() - start) * 1000)
    return sum(times) / len(times)  # ms per batch
```

## Common pitfalls

- Measuring latency on GPU with batch size 1, which artificially favors NAR models and ignores real-world batching overhead.
- Focusing only on peak throughput without accounting for per-sentence decoding latency or quality degradation under realistic CPU inference conditions.
- Using outdated or small test sets that do not reflect modern domain shifts or sentence length distributions.

## Evidence (verbatim from paper)

> A CTC-based NAR model achieves state-of-the-art BLEU on WMT14 but underperforms on recent test sets, highlighting the quality-speed trade-off and calling for more rigorous, standardized evaluation that includes both translation quality and realistic inference conditions.

## Citation

```bibtex
@misc{helcl2022nonautoregressive,
  title={Non-Autoregressive Machine Translation: It's Not as Fast as it Seems},
  author={Helcl et al. (2022)},
  year={2022},
  note={arXiv:2205.01966}
}
```

- arXiv: 2205.01966