Commt Mt Eval

commt-mt-eval

Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation — Yingfeng Luo et al. (arXiv:2503.06594, 2025)

What this evaluates

Evaluates machine translation quality across multiple language pairs and specialized tasks (general translation, terminology-constrained, and automatic post-editing). It measures how well encoder-decoder and decoder-only models generate accurate and fluent target sentences.

Datasets

• ComMT — total ?; splits: test (-1)

Metrics

• SacreBLEU (primary) — range: percent
  • Standard BLEU score computed using the SacreBLEU toolkit, typically with case-sensitive tokenization and no smoothing by default.
• COMET (wmt22-comet-da) — range: [0, 1]
  • Reference-free neural machine translation quality estimation model trained on WMT22 data, outputting a continuous score between 0 and 1.
• Terminology Success Rate (TSR) — range: percent
  • Percentage of required domain-specific terms that are correctly translated in the model output.
• Human Translation Edit Rate (HTER) — range: percent
  • Measures the amount of editing operations required to align a machine translation with a human reference translation, expressed as a percentage.

Input / output format

Input: Source language sentence (English, German, Czech, Russian, or Chinese).

Output: Target language translation sentence.

Scoring recipe

def compute_mt_metrics(predictions, references, sources=None):
    # SacreBLEU
    bleu = sacrebleu.corpus_bleu(predictions, [references])
    # COMET (requires source, prediction, reference)
    comet_scores = comet_model.predict(predictions, references, sources)
    # TSR & HTER require external terminology lists and edit logs
    return {
        'sacrebleu': bleu.score,
        'comet': float(comet_scores.mean())
    }

Common pitfalls

• SacreBLEU scores are highly sensitive to tokenization and case-sensitivity conventions; evaluators must use the exact SacreBLEU flags specified in the paper.
• COMET (wmt22-comet-da) requires the source sentence alongside predictions and references; omitting the source will cause evaluation failure or invalid scores.
• TSR and HTER require strict alignment between predicted terms and reference terminology; minor spelling variations or tokenization mismatches can drastically lower TSR.

Evidence (verbatim from paper)

We conducted tests on ComMT and evaluated the model’s translation performance across all tasks using COMET (wmt22-comet-da) (Rei et al., [2020]) and SacreBLEU (Post, [2018]). Additionally, we used Terminology Success Rate (TSR) for the terminology-constrained translation task and Human Translation Edit Rate (HTER) for the automatic post-editing task.

Citation

@misc{luo2025beyond,
  title={Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation},
  author={Yingfeng Luo et al.},
  year={2025},
  note={arXiv:2503.06594}
}

• arXiv: 2503.06594