Wmt Ape Eval

wmt-ape-eval

Learning to Copy for Automatic Post-Editing — Huang et al. (2019) (arXiv:1911.03627, 2019)

What this evaluates

Evaluates a model's ability to perform Automatic Post-Editing (APE) by correcting machine-translated German sentences using the original source text. It probes error detection, grammatical correction, and word-copying capabilities in a multi-source sequence-to-sequence setting.

Datasets

• WMT APE — total ?; splits: train (23000), dev2016 (1000), test2016 (2000), test2017 (2000), dev2018 (1000)

Metrics

• case-sensitive BLEU (primary) — range: [0, 100]
  • Standard n-gram precision with brevity penalty, computed using multi-bleu.perl. Evaluated case-sensitively as per WMT APE official guidelines.
• TER — range: percent
  • Translation Edit Rate: minimum number of edits (insertions, deletions, substitutions, shifts) required to change the hypothesis into the reference, divided by reference length. Calculated using TERcom.

Input / output format

Input: Pair of sequences: original source sentence (src) and machine-translated sentence (mt).

Output: Corrected German sentence (post-edit, pe).

Scoring recipe

bleu = multi_bleu.perl(predictions, references)  # case-sensitive
ter = TERcom(predictions, references)            # lower score is better
return {'bleu': bleu, 'ter': ter}

Common pitfalls

• BLEU is computed case-sensitively, unlike standard lowercased BLEU, which significantly impacts scores.
• The dataset distinguishes between PBSMT and NMT variants with different training sizes; results must be compared within the same task variant.
• Ensemble models report higher scores than single models; comparisons must specify which variant is used.

Evidence (verbatim from paper)

We used the same evaluation metrics as the official WMT APE task (Chatterjee et al., 2018): casesensitive BLEU and TER. BLEU is computed by multi-bleu.perl (Koehn et al., 2007). TER is calculated using TERcom.

Citation

@misc{huang2019learning,
  title={Learning to Copy for Automatic Post-Editing},
  author={Huang et al. (2019)},
  year={2019},
  note={arXiv:1911.03627}
}

• arXiv: 1911.03627