Wmt24++ Eval

wmt24++-eval

WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects — Deutsch et al. (2025) (arXiv:2502.12404, 2025)

What this evaluates

Evaluates machine translation systems across 55 languages and dialects using automatic metrics and significance testing to compare translation quality across different domains and language pairs.

Datasets

• WMT24++ — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: [0, 100] percent
  • Standard n-gram overlap metric between system output and reference translation, computed at corpus or sentence level.
• MetricX-24 — range: [0, 1]
  • Neural MT quality metric trained on WMT data; higher scores indicate better translation quality.
• XCOMET — range: [0, 1]
  • Cross-lingual COMET metric that predicts translation quality using a multilingual model; higher scores indicate better translations.
• COMETKiwi-23 — range: [0, 1]
  • Multilingual COMET variant optimized for cross-lingual evaluation; outputs a quality score where higher is better.
• Gemini-DA — range: [0, 1]
  • LLM-based direct assessment metric using Gemini; scores higher for better translation outputs.

Input / output format

Input: Source text, system-generated translation, and human reference translation (for reference-based metrics).

Output: Per-system scores per language, and significance-cluster-based rankings of systems per language.

Scoring recipe

for each language:
  scores = []
  for each system:
    if metric in ['BLEU', 'ChrF']:
      score = compute_reference_based_metric(system_output, reference)
    else:
      score = compute_neural_metric(system_output, source, reference)
    scores.append(score)
  # Exclude 38 bad source texts
  rankings = compute_significance_clusters(scores)
return rankings, scores

Common pitfalls

• Absolute metric values are not comparable across different languages; only relative rankings within a language are valid.
• System rankings are reported as significance clusters, not raw score orderings; systems in the same cluster are statistically indistinguishable.
• 38 bad source texts are explicitly excluded from all metric calculations and rankings.

Evidence (verbatim from paper)

Then, Table[3] contains an index for the figures with each metric’s system ranking and scores. The rankings and scores are calculated without the 38 bad source texts (see Appendix[C]). Note that due to the fact that absolute metric values are not comparable across languages, some MT systems may be favored if they only support languages for which the metric scores are high. | Metric | Ranking | Scores | BLEU | Figure[7] | Figure[10] | ChrF | Figure[7] | Figure[11] | MetricX-24 | Figure[7] | Figure[12] | XCOMET | Figure[8] | Figure[14] | COMETKiwi-23 | Figure[9] | Figure[16] | Gemini-DA | Figure[9] | Figure[17] |

Citation

@misc{deutsch2025wmt24++,
  title={WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects},
  author={Deutsch et al. (2025)},
  year={2025},
  note={arXiv:2502.12404}
}

• arXiv: 2502.12404