Flores101 Mt Eval

flores101-mt-eval

"A Little is Enough": Few-Shot Quality Estimation based Corpus Filtering improves Machine Translation — Batheja et al. (2023) (arXiv:2306.03507, 2023)

What this evaluates

Evaluates the translation quality of Neural Machine Translation (NMT) models trained on filtered pseudo-parallel corpora. It measures how well few-shot Quality Estimation (QE) based corpus filtering improves MT performance across low-resource and mid-resource language pairs compared to baselines and other filtering methods.

Datasets

• FLORES101 — total 1012; splits: test (1012)

Metrics

• BLEU (primary) — range: percent
  • Standard sentence-level BLEU score computed over the FLORES101 test set. It measures n-gram precision with a brevity penalty to penalize overly short translations.
• Pearson Correlation — range: [-1, 1]
  • Pairwise Pearson correlation coefficient computed between human-annotated Domain Adaptation (DA) quality scores and model-predicted quality scores on the QE test set.

Input / output format

Input: Source sentence in the source language (e.g., English, Marathi, Chinese, Hindi, or Bengali).

Output: Translated sentence in the target language.

Scoring recipe

def compute_bleu(predictions, references):
    # predictions: list of translated strings
    # references: list of reference strings (one per prediction)
    bleu_score = sacrebleu.corpus_bleu(predictions, [references])
    return bleu_score.score

Common pitfalls

• BLEU scores are reported separately for each translation direction (e.g., En→Mr vs Mr→En) and should not be averaged without explicit instruction.
• The FLORES101 test set contains exactly 1,012 sentence pairs per language pair, not the full 101 languages.
• QE filtering thresholds differ by language pair (-0.5 for En-Mr, -0.4 for Zh-En, 0 for Hi-Bn), so filtered corpus sizes vary significantly across methods and are not directly comparable.

Evidence (verbatim from paper)

For evaluation, we use the FLORES 101 test set which contains 1,012 sentence pairs for each language pair. Table 3: BLEU scores of En→Mr and Mr→En NMT models on FLORES101 test data.

Citation

@misc{batheja2023little,
  title={"A Little is Enough": Few-Shot Quality Estimation based Corpus Filtering improves Machine Translation},
  author={Batheja et al. (2023)},
  year={2023},
  note={arXiv:2306.03507}
}

• arXiv: 2306.03507