Wmt24 Mt Eval

wmt24-mt-eval

Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models — Domhan et al. (2025) (arXiv:2505.01761, 2025)

What this evaluates

Evaluates the ability of LLMs to accurately assess machine translation quality across varying input lengths (segment, document, and long-form). It probes whether LLMs can maintain consistent error detection and system ranking accuracy when processing longer texts, and tests prompting/fine-tuning strategies to mitigate length bias.

Datasets

• WMT'24 metrics shared task — total ?; splits: seg (-1), doc (-1), 5doc (-1)

Metrics

• system-level pairwise accuracy (primary) — range: [0, 1]
  • Measures the proportion of system pairs correctly ranked by the LLM compared to the ranking derived from human MQM annotations.
• character F1 score — range: [0, 1]
  • Computes precision and recall of predicted error spans against gold MQM spans at the character level, applying 0.5 partial credit for mismatches in error severity.
• number of error spans per document — range: other
  • Counts the total predicted MQM error spans identified by the LLM for each document.

Input / output format

Input: Source text and machine translation at varying granularities (segment, document, or concatenated 5-document blocks), accompanied by prompt templates (e.g., GEMBA, FSP, GMICL-5) containing few-shot demonstrations or direct assessment instructions.

Output: Structured MQM error span annotations (start/end offsets and severity) and/or direct quality scores, depending on the prompting strategy used.

Scoring recipe

# Pairwise Accuracy
human_rank = rank_systems_by_human_mqm(systems)
llm_rank = rank_systems_by_llm_error_counts(systems)
correct = sum(1 for i, j in pairs(systems) if (human_rank[i] > human_rank[j]) == (llm_rank[i] > llm_rank[j]))
pairwise_accuracy = correct / total_pairs

# Character F1
pred_spans = parse_llm_error_spans(output)
gold_spans = parse_gold_mqm(gold)
char_tp, char_fp, char_fn = compute_char_overlap(pred_spans, gold_spans, severity_penalty=0.5)
char_f1 = 2 * (char_precision * char_recall) / (char_precision + char_recall)

Common pitfalls

• LLMs exhibit strong length bias: longer inputs lead to fewer detected errors and degraded system ranking accuracy due to context limits and response length constraints.
• Providing additional segment-level demonstrations alone (e.g., GEMBA 3-shot) does not mitigate length bias; explicit alignment or granularity-matched prompting is required.
• Direct Assessment (DA) alone fails to improve ranking accuracy for long-form translations and must be combined with error-span prompting (FSP) to be effective.

Evidence (verbatim from paper)

We use system-level pairwise accuracy (Kocmi et al., 2021) as our evaluation metric. It measures the number of pairs of systems that are ranked correctly when compared to the ranking derived from human annotations. We use the official shared task scripts to access data and compute metrics. Additionally, we measure the number of error spans per document and the character F1 score. The latter is also used by the WMT shared task on quality estimation (Blain et al., 2023) and based on the precision/recall of error spans compared to gold annotations per character with partial credit (0.5) for a mismatch in error severity.

Citation

@misc{domhan2025sametokens,
  title={Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models},
  author={Domhan et al. (2025)},
  year={2025},
  note={arXiv:2505.01761}
}

• arXiv: 2505.01761