Comet Mt Eval

comet-mt-eval

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets — Yukhymenko et al. (2026) (arXiv:2602.22207, 2026)

What this evaluates

Probes the semantic fidelity and grammatical correctness of automated translation pipelines when converting English benchmarks into low-resource languages. It measures how well translation methods preserve task structure and downstream model performance consistency.

Datasets

• FLORES — total 3001; splits: devtest (1012)
• WMT24++ — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)

Metrics

• COMET (primary) — range: [0, 1]
  • Leverages multilingual pre-trained models to score translations by comparing source text, hypothesis, and reference. System-level scores are aggregated across all translations in the dataset.
• LLM-as-a-judge win rate — range: percent
  • Compares translated benchmarks against a baseline using a judge model (Gemini-2.5-Flash). Outputs win/draw/loss counts, aggregated as win rate or average improvement percentage.

Input / output format

Input: Per instance: English source sentence, human reference translation, and model-generated hypothesis translation. For benchmark evaluation: translated questions/answers paired with baseline translations for pairwise comparison.

Output: COMET score (float 0–1) per instance, aggregated to system-level mean. For LLM-as-judge: categorical preference (win/draw/loss) per pair.

Scoring recipe

scores = []
for src, ref, hyp in dataset:
    scores.append(comet_model.predict(src, ref, hyp))
return sum(scores) / len(scores)

wins, draws, losses = 0, 0, 0
for pair in benchmark_pairs:
    verdict = judge_model.compare(pair.baseline, pair.proposed)
    if verdict == 'win': wins += 1
    elif verdict == 'draw': draws += 1
    else: losses += 1
return wins / (wins + draws + losses)

Common pitfalls

• COMET scores are not absolute gold standards and can be biased by domain-specific training data or fail to capture stylistic preferences.
• Reference translations often contain multiple valid candidates, so high automated scores do not guarantee error-free or contextually perfect translations.
• LLM-as-a-judge evaluations may inherit model-family biases, as the judge model's architecture can influence preference scores.

Evidence (verbatim from paper)

We evaluate our proposed methods on English-Ukrainian translation using the COMET (Crosslingual Optimized Metric for Evaluation of Translation) metric. COMET leverages multilingual pre-trained models to assess translations by comparing source text, hypothesis, and reference, demonstrating higher correlation with human judgments than traditional metrics like BLEU or chrF++ (Rei et al., [2020]). We report COMET system-level scores (aggregated across all translations) in Table [1], using the Unbabel/XCOMET-XL model for reference-based quality estimation (Guerreiro et al., [2023]).

Citation

@misc{yukhymenko2026recovered,
  title={Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets},
  author={Yukhymenko et al. (2026)},
  year={2026},
  note={arXiv:2602.22207}
}

• arXiv: 2602.22207