Wmt24 Docmt Eval

wmt24-docmt-eval

Source-primed Multi-turn Conversation Helps Large Language Models Translate Documents — Hu et al. (2025) (arXiv:2503.10494, 2025)

What this evaluates

Evaluates document-level machine translation quality using multi-turn conversational prompting strategies with LLMs, measuring contextual coherence and translation accuracy across multiple language directions and domains.

Datasets

• WMT 24 General Track — total ?; splits: test (-1); repo https://github.com/ZurichNLP/multiturn-llm-docmt
• WMT 23 Chinese-to-English — total ?; splits: test (-1); repo https://github.com/ZurichNLP/multiturn-llm-docmt

Metrics

• dBLEU (primary) — range: [0, 100]
  • Document-level BLEU computed using sacreBLEU, considering n-gram matches across the entire translated document rather than per-segment.
• COMET-22 — range: [0, 1]
  • Neural translation quality metric using the COMET-22 default model. Due to context length limits, each segment is evaluated independently and scores are averaged.
• BlonDE — range: [0, 1]
  • Document-level metric measuring correctness of context-dependent features such as tense, pronouns, transliteration, entities, and connectives.

Input / output format

Input: Segmented source document paragraphs with prior conversation turns (context) and target language prompt. For the source-primed variant, the full source document is provided before translation begins.

Output: Translated target language paragraphs/documents.

Scoring recipe

# dBLEU (document-level)
dbleu = sacrebleu.corpus_bleu(predictions, [references], tokenize='none')

# COMET-22 (segment-level average)
comet_scores = []
for pred_seg, ref_seg in zip(predictions, references):
    comet_scores.append(comet22_model.predict(pred_seg, ref_seg))
avg_comet = sum(comet_scores) / len(comet_scores)

# BlonDE (document-level)
blonde_score = blonde_model.predict(doc_prediction, doc_reference)

Common pitfalls

• COMET-22 is evaluated per segment and averaged, which may underestimate document-level coherence due to its context window limit.
• BLEU is computed at the document level (dBLEU), not segment level, altering n-gram statistics and making direct comparison with sentence-level BLEU invalid.
• In-context learning (ICL) with 3 exemplars is used as a baseline setting, which can significantly boost scores compared to zero-shot prompting.

Evidence (verbatim from paper)

We use the COMET-22 default model Rei et al. ([2022]) and sacreBLEU Post ([2018]) implementation of BLEU to evaluate translation quality. For COMET, due to its context length limit, we evaluate each segment independently and report average scores; for BLEU, we consider n-gram matches at the document level Liu et al. ([2020]). Additionally, we use BlonDE Jiang et al. ([2022]) to evaluate document-level translation, which specifically measures the correctness of features that are known to benefit from wider context in Chinese-to-English translation, such as tense correctness, pronouns, transliteration, entities, and connectives.

Citation

@misc{hu2025sourceprimed,
  title={Source-primed Multi-turn Conversation Helps Large Language Models Translate Documents},
  author={Hu et al. (2025)},
  year={2025},
  note={arXiv:2503.10494}
}

• arXiv: 2503.10494