qhjqhj00 wmt2016-ape-eval
Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing — Junczys-Dowmunt et al. (2016) (arXiv:1605.04800, 2016)
What this evaluates
Evaluates Automatic Post-Editing (APE) systems by measuring how well they correct machine-translated German sentences using monolingual and bilingual neural translation models combined via log-linear weighting.
Datasets
- WMT 2016 APE Shared Task Development Set — total ?; splits: dev (-1)
Metrics
TER(primary) — range: percent- Translation Edit Rate: minimum number of edits (insertions, deletions, substitutions, shifts) to transform the hypothesis into the reference, normalized by reference length.
BLEU— range: percent- Bilingual Evaluation Understudy: geometric mean of n-gram precisions (1-4) with brevity penalty to penalize short translations.
Input / output format
Input: Machine-translated German sentence (for monolingual model) or English source sentence concatenated with MT output (for bilingual model).
Output: Post-edited German sentence.
Scoring recipe
def compute_ter(hypothesis, reference):
# Align hypothesis to reference using edit operations
edits = edit_distance_align(hypothesis, reference)
return (len(edits) / len(reference)) * 100
def compute_bleu(hypothesis, references):
# Compute n-gram precisions for n=1..4
precisions = [ngram_precision(hypothesis, ref, n) for n in range(1, 5)]
# Geometric mean with brevity penalty
bp = brevity_penalty(len(hypothesis), len(reference))
return bp * exp(mean(log(precisions))) * 100
Common pitfalls
- Model weights and PEP penalty are tuned on the development set, potentially inflating dev performance.
- Original post-editing data is oversampled 20x during training, which may bias the model toward the original MT errors rather than natural corrections.
- Faithfulness is enforced via a string-matching penalty during decoding, not as a post-hoc filter, which changes the generation dynamics.
Evidence (verbatim from paper)
We tune the weights on the development set towards lower TER scores; two iterations seem to be enough.
Citation
@misc{junczysdowmunt2016loglinear,
title={Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing},
author={Junczys-Dowmunt et al. (2016)},
year={2016},
note={arXiv:1605.04800}
}
- arXiv: 1605.04800