Wmt21 News Translation Eval

wmt21-news-translation-eval

Non-Autoregressive Machine Translation: It's Not as Fast as it Seems — Helcl et al. (2022) (arXiv:2205.01966, 2022)

What this evaluates

Evaluates the translation quality and inference speed of non-autoregressive versus autoregressive machine translation models on English-German news text. It probes the practical trade-offs between decoding latency and translation accuracy under realistic deployment conditions.

Datasets

• WMT21 News Translation — total ?; splits: train (-1), test (-1)

Metrics

• BLEU (primary) — range: [0, 100]
  • Standard n-gram overlap metric between predicted and reference translations, typically computed with sentence-level averaging and smoothing.
• Latency — range: ms or words/sec
  • Inference time measured in milliseconds per sentence or words per second, evaluated under varying batch sizes and hardware conditions (CPU vs GPU).

Input / output format

Input: English source sentence (string)

Output: German target sentence (string)

Scoring recipe

def compute_bleu(predictions, references):
    return nltk.translate.bleu_score.corpus_bleu(references, predictions) * 100

def compute_latency(model, inputs, batch_size=1, device='cpu'):
    times = []
    for batch in get_batches(inputs, batch_size):
        start = time.time()
        _ = model.generate(batch, device=device)
        times.append((time.time() - start) * 1000)
    return sum(times) / len(times)  # ms per batch

Common pitfalls

• Measuring latency on GPU with batch size 1, which artificially favors NAR models and ignores real-world batching overhead.
• Focusing only on peak throughput without accounting for per-sentence decoding latency or quality degradation under realistic CPU inference conditions.
• Using outdated or small test sets that do not reflect modern domain shifts or sentence length distributions.

Evidence (verbatim from paper)

A CTC-based NAR model achieves state-of-the-art BLEU on WMT14 but underperforms on recent test sets, highlighting the quality-speed trade-off and calling for more rigorous, standardized evaluation that includes both translation quality and realistic inference conditions.

Citation

@misc{helcl2022nonautoregressive,
  title={Non-Autoregressive Machine Translation: It's Not as Fast as it Seems},
  author={Helcl et al. (2022)},
  year={2022},
  note={arXiv:2205.01966}
}

• arXiv: 2205.01966