Wmt Bleu Eval

wmt-bleu-eval

Unsupervised Parallel Corpus Mining on Web Data — Lai et al. (2020) (arXiv:2009.08595, 2020)

What this evaluates

Evaluates the quality of an unsupervised web-mined parallel corpus by training neural machine translation models on it and measuring translation performance on standard WMT test sets. It probes whether mined pseudo-parallel data can effectively substitute for human-labeled data in both supervised and unsupervised MT training pipelines.

Datasets

• WMT2014 test set — total ?; splits: test (-1)
• WMT2016 test set — total ?; splits: test (-1)

Metrics

• BELU (primary) — range: [0, 100]
  • Standard corpus-level BLEU score with 4-gram precision and brevity penalty. The paper consistently misspells the metric as 'BELU' in the text and tables.

Input / output format

Input: Source sentences from WMT2014/WMT2016 test sets (English, French, German, Romanian).

Output: Target language translations generated by the NMT model.

Scoring recipe

def compute_bleu(predictions, references):
    # predictions and references are lists of tokenized sentences
    # Uses standard corpus-level BLEU with 4-grams and brevity penalty
    bleu_score = corpus_bleu(references, predictions, weights=(0.25, 0.25, 0.25, 0.25))
    return bleu_score * 100  # Scale to percentage

Common pitfalls

• The paper consistently misspells 'BLEU' as 'BELU' in the text and tables, which may cause confusion when searching for the metric.
• The evaluation mixes supervised (Scaling NMT) and unsupervised (XLM/MASS) training protocols but reports the same metric, making it unclear whether back-translation augmentation was applied for each reported score.
• The ablation study shows raw (unfiltered) data performs similarly to filtered data, which may mislead readers about the filtration step's necessity despite the classifier successfully removing low-quality pairs.

Evidence (verbatim from paper)

We follow the experiment setting in the Scaling NMT paper (Ott et al., 2018), including model architecture and choice of the hyper-parameters, and report the BELU score on the En-Fr and En-De directions on the WMT2014 test sets.

Citation

@misc{lai2020unsupervised,
  title={Unsupervised Parallel Corpus Mining on Web Data},
  author={Lai et al. (2020)},
  year={2020},
  note={arXiv:2009.08595}
}

• arXiv: 2009.08595