Wmt Mt Eval

wmt-mt-eval

Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation — Haoran Xu et al. (2024) (arXiv:2401.08417, 2024)

What this evaluates

This protocol evaluates the machine translation quality of large language models across multiple language pairs. It measures translation accuracy and fluency by comparing model outputs against gold references and state-of-the-art baselines using neural quality estimation metrics. The benchmark probes the model's ability to generalize across diverse language directions and avoid generating near-perfect but flawed translations.

Datasets

• WMT'21 Test Set — total ?; splits: test (-1)
• WMT'22 Test Set — total ?; splits: test (-1)
• WMT'23 Test Set — total ?; splits: test (-1)

Metrics

• KIWI-XXL (primary) — range: [0, 100]
  • A reference-free neural machine translation quality estimation metric that predicts human judgment scores. It takes a source sentence and a hypothesis translation as input and outputs a continuous score reflecting translation quality.
• XCOMET — range: [0, 100]
  • A reference-free neural metric that estimates translation quality by leveraging multilingual representations and cross-lingual alignment. It outputs a score correlating with human preference.
• KIWI-22 — range: [0, 100]
  • A smaller, reference-free quality estimation model (Unbabel/wmt22-cometkiwi-da) that predicts human judgment scores for translation pairs.

Input / output format

Input: Source sentence in a source language (e.g., English, German, Czech, Icelandic, Chinese, Russian).

Output: Translated sentence in the target language.

Scoring recipe

def compute_mt_scores(predictions, references, sources, metric_name):
    scores = []
    for src, ref, pred in zip(sources, references, predictions):
        if metric_name in ['KIWI-XXL', 'XCOMET', 'KIWI-22']:
            # Reference-free neural metric predicts human judgment score
            score = metric_model.predict(source=src, hypothesis=pred)
        else:
            # Reference-based metric (e.g., sacreBLEU, COMET-22)
            score = metric_model.predict(reference=ref, hypothesis=pred)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Relying on gold human references for evaluation, as the paper explicitly argues they are often inferior to system-generated translations and can compromise evaluation reliability.
• Comparing against models that were trained on the specific test set being evaluated (e.g., TowerInstruct used WMT'22 data, so it must be excluded from WMT'22 comparisons).
• Averaging scores across different metrics without noting that KIWI-22, KIWI-XXL, and XCOMET have different scales and calibration, though the paper reports them separately and averages across directions.

Evidence (verbatim from paper)

Our emphasis is primarily on reference-free evaluation models, due to our analysis in Section [2], which questions the reliability of gold references and highlights that evaluations can be compromised by poor-quality references (Kocmi et al., [2023]; Freitag et al., [2023]). The reference-free models used for evaluation include KIWI-XXL, XCOMET, and a smaller yet popular model, Unbabel/wmt22-cometkiwi-da (hereinafter referred to as KIWI-22).

Citation

@misc{xu2024contrastive,
  title={Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation},
  author={Haoran Xu et al. (2024)},
  year={2024},
  note={arXiv:2401.08417}
}

• arXiv: 2401.08417