Wmt17 Mt Eval

wmt17-mt-eval

The University of Edinburgh's Neural MT Systems for WMT17 — Rico Sennrich et al. (WMT17 / arXiv:1708.00726, 2017)

What this evaluates

Evaluates neural machine translation systems across multiple language pairs in news and biomedical domains. Probes translation quality, domain adaptation, and system combination techniques like ensembling and reranking on held-out parallel test sets.

Datasets

• WMT17 News Task — total ?; splits: dev (-1), test (-1)
• HimL Biomedical Task — total ?; splits: tuning (-1), test (-1)

Metrics

• BLEU (primary) — range: percent
  • Sentence-level BLEU with 4-gram precision, geometric mean, and brevity penalty. Scores are reported as raw values (e.g., 30.9) representing the percentage scale.

Input / output format

Input: Source language sentence, preprocessed with language-specific tokenization (e.g., Jieba for Chinese, custom tokenizer for Latvian) and BPE/subword segmentation.

Output: Target language sentence, postprocessed to remove extra spaces and convert ASCII punctuation to appropriate CJK Unicode equivalents where applicable.

Scoring recipe

def compute_bleu(predictions, references):
    precisions = []
    for n in range(1, 5):
        matches = sum(min(count_ngram(pred, n), count_ngram(ref, n)) for pred, ref in zip(predictions, references))
        total = sum(count_ngram(pred, n) for pred in predictions)
        precisions.append(matches / total if total > 0 else 0)
    bp = math.exp(1 - len(references)/len(predictions)) if len(predictions) < len(references) else 1
    return bp * math.exp(sum(math.log(p) for p in precisions if p > 0) / 4) * 100

Common pitfalls

• Final scores rely on complex system combination: ensembling left-to-right and right-to-left models, followed by reranking, not single-model inference.
• Length penalty (alpha) is tuned per language pair on the dev set (e.g., 0.6 for EN→LV, 0.7 for LV→EN) and significantly impacts BLEU.
• Biomedical domain adaptation requires careful synthetic data generation and diacritic normalization, which can introduce errors if not handled correctly.
• Back-translation data quality and filtering heavily influence performance, especially for low-resource or morphologically complex pairs.

Evidence (verbatim from paper)

The final system is an ensemble of the best validation BLEU model from each of the three target left-right runs, rescored with the three target right-left runs, and reranked.

Citation

@misc{sennrich2017wmt17,
  title={The University of Edinburgh's Neural MT Systems for WMT17},
  author={Rico Sennrich et al.},
  year={2017},
  note={WMT17 / arXiv:1708.00726}
}

• arXiv: 1708.00726