Wmt20 Zh En Eval

wmt20-zh-en-eval

Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand — First Author et al. (2021) (arXiv:2112.04139, 2021)

What this evaluates

Evaluates machine translation quality by scoring system-generated English translations against human reference sentences and expert MQM scores. It measures how well automatic metrics correlate with human judgments across different translation systems.

Datasets

• WMT20 ZH-EN — total ?; splits: test (-1)

Metrics

• COMET (primary) — range: [0, 1]
  • A neural reference-based metric trained to predict human judgments. Scores are typically normalized to [0, 1].
• COMET-QE — range: [0, 1]
  • A referenceless variant of COMET that uses only the source sentence and hypothesis to predict quality.
• BLEURT — range: [0, 1]
  • A BERT-based metric fine-tuned on human judgments specifically for in-English translation evaluation.
• Prism-ref — range: [0, 1]
  • A probabilistic reference-based metric that scores translations using a language model over the reference.

Input / output format

Input: Source sentence in Chinese and reference sentence(s) (Human-A and Human-B).

Output: Generated English translation sentence.

Scoring recipe

scores = []
for src, ref, hyp in dataset:
    metric_score = compute_metric(hyp, ref)
    human_score = average(expert_MQM_scores[hyp])
    scores.append((metric_score, human_score))
return pearson_correlation([s[0] for s in scores], [s[1] for s in scores])

Common pitfalls

• Using only a single human reference instead of all available references (Human-A and Human-B) reduces correlation with human judgments.
• Failing to detokenize model outputs before scoring can artificially lower reference-based metric scores.

Evidence (verbatim from paper)

Table 1: Summary of Billboards as of Jan. 10, 2022. ... Top Metric: COMET ... Ensemble of Metrics: 1.72·COMET-QE+1.48·COMET+1.21·BLEURT ... We use all human translations available as a reference set for reference-based metrics. Concretely, every test instance in WMT20 ZH-EN has two translations provided by different human translation services: Human-A and Human-B... Each output sentence is evaluated by three professional translators. Following Freitag et al. (2021), the three scores are averaged to get an instance-level score.

Citation

@misc{first2021billboard,
  title={Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand},
  author={First Author et al. (2021)},
  year={2021},
  note={arXiv:2112.04139}
}

• arXiv: 2112.04139