Wmt22 Ted Translation Eval

wmt22-ted-translation-eval

Exploring Representational Disparities Between Multilingual and Bilingual Translation Models — Verma et al. (2023) (arXiv:2305.14230, 2023)

What this evaluates

Evaluates machine translation models on English-to-multiple-target-language pairs using standard development sets. It measures translation quality via BLEU scores to compare bilingual versus multilingual decoder representations and capacity.

Datasets

• WMT22 General Machine Translation — total ?; splits: train (-1), dev (-1)
• Multitarget TED talks — total ?; splits: train (-1), dev (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric for machine translation, computed using sacrebleu. It measures the geometric mean of modified n-gram precisions (typically up to 4-grams) with a brevity penalty.

Input / output format

Input: Source sentence in English, optionally prepended with a target language ID token for multilingual models.

Output: Generated target language sentence.

Scoring recipe

import sacrebleu
# predictions: list of generated target sentences
# references: list of gold target sentences
bleu_score = sacrebleu.corpus_bleu(predictions, [references]).score

Common pitfalls

• Evaluation is performed on development sets rather than held-out test sets, which may not reflect final generalization performance.
• Checkpoint selection criteria differ across scales: small/TED models use best validation BLEU after 80 epochs, while large models use average validation loss after 240k updates, potentially confounding metric comparisons.
• Vocabulary size differs between bilingual (16K) and multilingual (32K) setups, meaning BLEU differences may reflect tokenization capacity rather than representational isotropy.

Evidence (verbatim from paper)

We report BLEU scores on our dev sets computed with sacrebleu Papineni et al. (2002); Post (2018).

Citation

@misc{verma2023representational,
  title={Exploring Representational Disparities Between Multilingual and Bilingual Translation Models},
  author={Verma et al. (2023)},
  year={2023},
  note={arXiv:2305.14230}
}

• arXiv: 2305.14230