Mnmt Eval

mnmt-eval

Parameter Differentiation based Multilingual Neural Machine Translation — Qian Wang et al. (2021) (arXiv:2112.13619, 2021)

What this evaluates

Evaluates multilingual neural machine translation performance across many-to-one, one-to-many, and many-to-many translation scenarios, testing how dynamic parameter differentiation impacts translation quality across diverse language pairs and resource levels.

Datasets

• OPUS — total ?; splits: train (-1), val (-1)
• WMT — total ?; splits: train (-1)
• IWSLT'17 — total ?; splits: train (-1)

Metrics

• BLEU (primary) — range: [0, 100]
  • Standard n-gram precision score computed using SacreBLEU, which handles tokenization and reference formatting automatically.

Input / output format

Input: Source sentence in a source language (e.g., English, German, etc.)

Output: Target sentence in the target language

Scoring recipe

import sacrebleu
def compute_bleu(predictions, references):
    return sacrebleu.corpus_bleu(predictions, [references]).score

Common pitfalls

• Using a different BLEU implementation (e.g., Moses or custom tokenization) instead of SacreBLEU will yield non-comparable scores.
• Forgetting to apply the temperature-based sampling (τ=5) during WMT dataset training, which is explicitly required for fair comparison.
• Comparing model sizes without normalizing for the parameter differentiation upper bound (O), as the method's size scales with the differentiation limit rather than language count.

Evidence (verbatim from paper)

We measure the translation quality by BLEU score (Papineni et al. 2002) with SacreBLEU. We use the public OPUS and WMT multilingual datasets to evaluate our method on many-to-one (M2O) and one-to-many (O2M) translation scenarios, and the IWSLT datasets for the many-to-many (M2M) translation scenario.

Citation

@misc{wang2021parameter,
  title={Parameter Differentiation based Multilingual Neural Machine Translation},
  author={Qian Wang et al. (2021)},
  year={2021},
  note={arXiv:2112.13619}
}

• arXiv: 2112.13619