Flores200 Mt Eval

flores200-mt-eval

Beyond English: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs — Luo et al. (2025) (arXiv:2511.07003, 2025)

What this evaluates

Evaluates multilingual machine translation quality across 60 languages and 234 translation directions. It specifically probes a model's ability to handle high-, medium-, and low-resource languages while mitigating directional degeneration in symmetric multi-way translation.

Datasets

• FLORES-200 — total 2500; splits: devtest (2500); HF flores

Metrics

• COMET-22 (primary) — range: percent (0-100)
  • A neural reference-based metric that uses a multilingual encoder (XLM-R) to compute contextual representations of source, reference, and hypothesis, followed by a cross-attention mechanism and a regression head to predict a quality score. Scores are scaled to a 0-100 range for reporting.

Input / output format

Input: Source sentence in language X, optionally prefixed with explicit direction tags and a target-language separator.

Output: Target sentence in language Y.

Scoring recipe

import comet
model = comet.load_model('Unbabel/comet-22')
scores = []
for src, ref, hyp in zip(sources, references, hypotheses):
    score = model.predict({'src': src, 'ref': ref, 'hyp': hyp})
    scores.append(score * 100)
return sum(scores) / len(scores)

Common pitfalls

• Evaluating on language pairs not supported by both the baseline and the proposed model, which skews cross-model comparisons.
• Overlooking directional asymmetry, as performance often differs significantly between forward (En/Zh→X) and reverse (X→En/Zh) translation directions.
• Ignoring the custom Chinese-Mongolian testset, which was created via native annotators to fill a gap in FLORES-200.

Evidence (verbatim from paper)

We evaluate on FLORES-200 devtest. We adopt COMET-22 as our primary evaluation metric, and report SacreBLEU in the Appendix. For fairness, evaluation is conducted on the intersection of language pairs supported by each baseline and LMT, covering Chinese-centric directions when applicable.

Citation

@misc{luo2025beyondenglish,
  title={Beyond English: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2511.07003}
}

• arXiv: 2511.07003