Nmt Translation Eval

nmt-translation-eval

Data Diversification: A Simple Strategy For Neural Machine Translation — Nguyen et al. (2019) (arXiv:1911.01986, 2019)

What this evaluates

This evaluation probes a neural machine translation model's ability to translate sentences across multiple language pairs, covering high-resource (English-German, English-French) and low-resource (English-Nepali, English-Sinhala) settings. It measures translation quality using BLEU scores to assess the effectiveness of data diversification strategies without requiring monolingual data or additional parameters.

Datasets

• WMT'14 — total ?; splits: train (-1), val (-1), test (-1)
• IWSLT'13/14 — total ?; splits: train (-1), val (-1), test (-1)
• Low-resource (Guzmán et al.) — total ?; splits: dev (-1), test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric between predicted and reference translations. For low-resource tasks, from-English pairs use tokenized BLEU, while to-English pairs use detokenized SacreBLEU. High-resource tasks use standard tokenized BLEU.

Input / output format

Input: Source sentence in the source language (e.g., English, German, Nepali, Sinhala).

Output: Target sentence in the target language.

Scoring recipe

def compute_bleu(predictions, references):
    # Tokenize if from-English (low-resource)
    # Detokenize if to-English (low-resource)
    # Standard tokenized BLEU for WMT/IWSLT
    return corpus_bleu(references, predictions)

Common pitfalls

• Mixing tokenized and detokenized BLEU scoring conventions between from-English and to-English low-resource tasks.
• Failing to filter duplicate pairs when augmenting training data with synthetic translations, which inflates dataset size metrics.
• Assuming monolingual data is used for augmentation, whereas the method explicitly avoids it.

Evidence (verbatim from paper)

We conduct experiments on the standard WMT'14 English-German (En-De) and English-French (En-Fr) translation tasks. The training datasets contain about 4.5M and 35M sentence pairs respectively. ... We use newstest2013 as the development set, and newstest2014 for testing. ... From the results on WMT newtest2014 testset in Table 2, we observe that the scale Transformer [18], which originally gives 29.3 BLEU in the En-De task, now gives 30.7 BLEU with our data diversification strategy... As done by Guzmán et al. [11], the from-English pairs are measured in tokenized BLEU, while to-English are measured in detokenized SacreBLEU.

Citation

@misc{nguyen2019datadiversification,
  title={Data Diversification: A Simple Strategy For Neural Machine Translation},
  author={Nguyen et al. (2019)},
  year={2019},
  note={arXiv:1911.01986}
}

• arXiv: 1911.01986