Wikimatrix Eval

wikimatrix-eval

WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia — Schwenk et al. (2019) (arXiv:1907.05791, 2019)

What this evaluates

Assesses the quality of automatically mined parallel sentence pairs by training neural machine translation models and measuring their downstream translation accuracy.

Datasets

• WikiMatrix — total 135000000; splits: train (-1); repo https://github.com/facebookresearch/LASER

Metrics

• BLEU (primary) — range: percent
  • Standard BLEU score (Papineni et al., 2002) computed on the TED test set. It measures the geometric mean of 1- to 4-gram precisions with a brevity penalty to penalize overly short translations.

Input / output format

Input: Source sentences from the TED test set.

Output: Translated target sentences generated by the trained NMT model.

Scoring recipe

def compute_bleu(predictions, references):
    # Standard BLEU-4 calculation with brevity penalty
    return nltk.translate.bleu_score.corpus_bleu(
        [[ref] for ref in references],
        predictions,
        weights=(0.25, 0.25, 0.25, 0.25)
    ) * 100

Common pitfalls

• Evaluation is only reported for language pairs with >25k mined sentences, not the full 1620 pairs.
• The metric measures downstream NMT translation quality rather than direct bitext alignment accuracy.

Evidence (verbatim from paper)

We then turn to a qualitative assessment by training NMT systems for all language pairs with more than twenty-five thousand mined sentences (Section 5.2). Evaluated via NMT on TED, the mined data achieves strong BLEU scores, demonstrating effectiveness for training machine translation systems in low-resource and non-English language settings.

Citation

@misc{schwenk2019wikimatrix,
  title={WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia},
  author={Schwenk et al. (2019)},
  year={2019},
  note={arXiv:1907.05791}
}

• arXiv: 1907.05791