Ccmatrix Eval

ccmatrix-eval

CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB — Schwenk et al. (2019) (arXiv:1911.04944, 2019)

What this evaluates

Evaluates the quality of mined parallel sentence pairs by training machine translation systems on them and measuring translation performance. It probes the effectiveness of global, margin-based bitext mining in a multilingual embedding space. The benchmark measures how well the mined data generalizes across different language families and scripts.

Datasets

• CCMatrix — total 4500000000; splits: train (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric for machine translation, typically computed over tokenized text. Values are usually reported as percentages.

Input / output format

Input: Monolingual sentences in two languages from a curated Common Crawl corpus.

Output: Aligned parallel sentence pairs (bitexts) used to train NMT systems, evaluated via BLEU on TED test sets.

Scoring recipe

# Mine bitexts using LASER embeddings with margin threshold 1.06
bitexts = mine_parallel_sentences(monolingual_corpus, threshold=1.06)
# Train NMT model on mined bitexts
model = train_nmt(bitexts)
# Evaluate on TED test set
predictions = model.translate(ted_test_set)
bleu_score = compute_bleu(predictions, ted_test_set.golds)

Common pitfalls

• Higher margin thresholds improve alignment quality but drastically reduce dataset size, requiring a careful trade-off.
• Computational complexity limits the number of languages covered, so results may not generalize to all language families equally.

Evidence (verbatim from paper)

The margin threshold used to mine parallel sentences will impact the quality of produced bitexts. A higher threshold will lead to better aligned sentences, and thus higher quality bitexts, but also to smaller datasets. Thus, there is a trade-off between the size of the extracted bitexts, and their quality. Exploratory experiments showed that a threshold around 1.06 seems to give good results. To confirm this, we trained and evaluated machine translation systems on the Hu-Da pair for different values of the treshold. We report results in Fig. 4, showing that 1.06 leads to the best performance. CCMatrix: alignments with English. We give the number of the monolingual texts and the extracted parallel sentences (all numbers in millions) for a margin threshold of 1.06, as well as the BLEU scores on the TED test.

Citation

@misc{schwenk2019ccmatrix,
  title={CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB},
  author={Schwenk et al. (2019)},
  year={2019},
  note={arXiv:1911.04944}
}

• arXiv: 1911.04944