Wmt2016 Mt Eval

wmt2016-mt-eval

Edinburgh Neural Machine Translation Systems for WMT 16 — Sennrich et al. (2016) (arXiv:1606.02891, 2016)

What this evaluates

Evaluates neural machine translation systems across multiple language pairs (EN-DE, EN-CS, CS-EN, EN-RO, RO-EN, EN-RU, RU-EN) on news text. It measures translation quality using BLEU scores on held-out test sets to assess the impact of techniques like back-translation, ensembling, and subword segmentation.

Datasets

• WMT 2016 News Translation — total ?; splits: test (-1), dev (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard corpus-level BLEU score measuring 4-gram precision with a brevity penalty. Reported as a percentage (0-100) in the tables.

Input / output format

Input: Source language sentence, typically tokenized and encoded with BPE subword operations (or raw text for baselines).

Output: Target language sentence, typically tokenized and encoded with BPE subword operations (or raw text for baselines).

Scoring recipe

def compute_bleu(hypotheses, references):
    # hypotheses: list of str (model outputs)
    # references: list of list of str (gold translations)
    # Uses standard corpus-level BLEU with 4-gram precision and brevity penalty
    bleu_score = corpus_bleu(references, hypotheses)
    return round(bleu_score * 100, 1)

Common pitfalls

• BLEU scores are highly sensitive to tokenization and BPE segmentation; results are not directly comparable across different preprocessing pipelines or character-level baselines.
• The paper reports both dev and test scores, but test scores are on held-out data (newstest2016) and must not be used for hyperparameter tuning or model selection.
• Ensemble and reranking results aggregate multiple model checkpoints, which can mask per-checkpoint variance and inflate reported gains compared to single-model baselines.

Evidence (verbatim from paper)

Table 2: English↔German translation results (Bleu) on dev (newstest2015) and test (newstest2016). Submitted system in bold.

Citation

@misc{sennrich2016edinburgh,
  title={Edinburgh Neural Machine Translation Systems for WMT 16},
  author={Sennrich et al. (2016)},
  year={2016},
  note={arXiv:1606.02891}
}

• arXiv: 1606.02891