Wmt18 News Eval

wmt18-news-eval

Machine Translation Customization via Automatic Training Data Selection from the Web — Vu et al. (2021) (arXiv:2102.10243, 2021)

What this evaluates

Evaluates the effectiveness of automatic web data selection for domain-specific machine translation in the news domain. It probes whether a document-level topic classifier can filter noisy parallel data to improve MT system performance compared to state-of-the-art baselines on the WMT-18 benchmark.

Datasets

• WMT-18 News Shared Task — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric between system translations and reference translations, computed using the sacrebleu toolkit with default WMT settings (case-insensitive, tokenized with 13a).
• Accuracy — range: percent
  • Proportion of correctly classified document batches (news vs. non-news) by the topic classifier.
• Average z-score — range: other
  • Human evaluation metric where individual assessor scores are standardized by their mean and standard deviation, then averaged across assessors for each system.

Input / output format

Input: Source language sentences or document batches (English or German) provided as parallel text pairs for translation.

Output: Target language translations (German or English) generated by the MT model.

Scoring recipe

# Compute BLEU using sacrebleu as specified in the paper
import sacrebleu

def compute_bleu(predictions, references):
    # predictions: list of system translations (str)
    # references: list of reference translations (list of str or list of list of str)
    score = sacrebleu.corpus_bleu(predictions, [references])
    return score.score

# Compute Accuracy for classifier
# predictions: list of predicted labels (0/1)
# gold: list of true labels (0/1)
accuracy = sum(p == g for p, g in zip(predictions, gold)) / len(gold)

# Compute Average z-score for human eval
# scores: list of raw assessor scores per system
# mean, std: assessor-level mean and std
z_scores = [(s - mean) / std for s in scores]
avg_z = sum(z_scores) / len(z_scores)

Common pitfalls

• Using sentence-level classification instead of document/batch-level classification significantly degrades classifier accuracy and data selection quality.
• Computing BLEU without sacrebleu or with different tokenization standards leads to non-comparable scores against the reported WMT-18 baselines.
• Comparing model performance without accounting for differences in training data volume, model architecture (e.g., Transformer-Base vs Big), or use of back-translation.

Evidence (verbatim from paper)

We evaluated such models against the standard WMT-2017 and WMT-2018 test sets, using BLEU as our evaluation metric. The results are reported in Table 3, under the column Buckets, using the evaluation tool, sacrebleu [17].

Citation

@misc{vu2021mtcustomization,
  title={Machine Translation Customization via Automatic Training Data Selection from the Web},
  author={Vu et al. (2021)},
  year={2021},
  note={arXiv:2102.10243}
}

• arXiv: 2102.10243