Opus 100 Nmt Eval

opus-100-nmt-eval

Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation — Zhang et al. (2020) (arXiv:2004.11867, 2020)

What this evaluates

Evaluates massively multilingual neural machine translation models on translation quality and language accuracy across 100 languages, including zero-shot translation between unseen language pairs.

Datasets

• OPUS-100 — total ?; splits: train (-1), test (-1); repo https://github.com/EdinburghNLP/opus-100-corpus

Metrics

• BLEU_94 (primary) — range: percent
  • Average SacreBLEU score computed over all 94 language pairs that have dedicated test sets in the one-to-many or many-to-many setting.
• BLEU_zero — range: percent
  • Average SacreBLEU score computed over all zero-shot translation directions (unseen X→Y pairs) in the test set.
• ACC_zero — range: percent
  • Translation-language accuracy measuring the proportion of generated outputs that are correctly identified as the target language using the langdetect library.
• WR — range: percent
  • Win ratio counting the proportion of tasks or language pairs where the proposed approach outperforms the baseline model.

Input / output format

Input: Source sentence in language X, prefixed with a language token specifying the target language (e.g., for English).

Output: Translated sentence in the specified target language.

Scoring recipe

bleu_scores = [sacrebleu.corpus_bleu(pred, [ref]).score for pred, ref in zip(predictions, references)]
metric_value = sum(bleu_scores) / len(bleu_scores)

correct = 0
for pred, target_lang in zip(predictions, target_languages):
    detected_lang = langdetect.detect(pred)
    if detected_lang == target_lang:
        correct += 1
acc_zero = correct / len(predictions)

Common pitfalls

• Off-target translation severely degrades zero-shot BLEU scores, making language accuracy a critical diagnostic metric.
• Training data is highly imbalanced across the 100 languages, causing English→X performance to lag behind X→English.
• ROBT improves zero-shot translation but slightly sacrifices in-target translation BLEU scores.

Evidence (verbatim from paper)

We adopt BLEU (Papineni et al., 2002) for translation evaluation with the toolkit SacreBLEU (Post, 2018). We employ the langdetect library to detect the language of translations, and measure the translation-language accuracy for zero-shot cases. Rather than providing numbers for each language pair, we report average BLEU over all 94 language pairs with test sets (BLEU_94).

Citation

@misc{zhang2020improving,
  title={Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation},
  author={Zhang et al. (2020)},
  year={2020},
  note={arXiv:2004.11867}
}

• arXiv: 2004.11867