Wmt20 En De Eval

wmt20-en-de-eval

Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand — First Author et al. (2021) (arXiv:2112.04139, 2021)

What this evaluates

Evaluates machine translation quality by scoring system-generated German translations against human reference sentences and expert MQM scores. It measures how well automatic metrics correlate with human judgments across different translation systems.

Datasets

• WMT20 EN-DE — total ?; splits: test (-1)

Metrics

• COMET (primary) — range: [0, 1]
  • A neural reference-based metric trained to predict human judgments. Scores are typically normalized to [0, 1].
• COMET-QE — range: [0, 1]
  • A referenceless variant of COMET that uses only the source sentence and hypothesis to predict quality.
• Prism-ref — range: [0, 1]
  • A probabilistic reference-based metric that scores translations using a language model over the reference.

Input / output format

Input: Source sentence in English and reference sentence(s) (Human-A, Human-B, and Human-P).

Output: Generated German translation sentence.

Scoring recipe

scores = []
for src, ref, hyp in dataset:
    metric_score = compute_metric(hyp, ref)
    human_score = average(expert_MQM_scores[hyp])
    scores.append((metric_score, human_score))
return pearson_correlation([s[0] for s in scores], [s[1] for s in scores])

Common pitfalls

• Ignoring the Human-P paraphrased reference, which mitigates translationese effects and diversifies the reference set.
• Not accounting for the arbitrary distinction between Human-A and Human-B when swapping roles for mixed-effects analysis.

Evidence (verbatim from paper)

Table 1: Summary of Billboards as of Jan. 10, 2022. ... Top Metric: COMET ... Ensemble of Metrics: 1.19·COMET+0.36·COMET-QE+0.02·Prism-ref ... In addition to Human-A and Human-B, WMT20 EN-DE provides a translation that is created by linguists who are asked to paraphrase Human-A and Human-B as much as possible (Human-P)... These paraphrased translations are shown to increase correlations with human judgments by mitigating the translationese effect and diversifying the reference when the generation quality is measured by reference-based metrics

Citation

@misc{first2021billboard,
  title={Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand},
  author={First Author et al. (2021)},
  year={2021},
  note={arXiv:2112.04139}
}

• arXiv: 2112.04139