qhjqhj00 wmt21-nmt-eval
NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21 — Subramanian et al. (2021) (arXiv:2111.08634, 2021)
What this evaluates
Evaluates neural machine translation performance across news and biomedical domains for English-German and English-Russian language pairs. It probes the model's ability to handle domain-specific vocabulary, cross-lingual alignment, and translation quality under constrained data conditions typical of shared task tracks.
Datasets
- WMT21 News & Biomedical Shared Tasks — total ?; splits: test (-1)
Metrics
BLEU(primary) — range: percent- Standard corpus-level BLEU score computed over 1-4 gram precisions with a brevity penalty. Calculated using the official WMT sacreBLEU script with Moses tokenization and lowercasing conventions.
Input / output format
Input: Source sentence in English, German, or Russian.
Output: Translated sentence in the target language.
Scoring recipe
def compute_bleu(predictions, references):
# Normalize text (lowercase, Moses tokenize)
# Compute 1-4 gram precisions for each prediction-reference pair
# Apply brevity penalty based on length ratio
# Return geometric mean of precisions * brevity_penalty
return sacrebleu.corpus_bleu(predictions, [references])
Common pitfalls
- Domain-specific vocabulary filtering thresholds (e.g., 0.90 cutoff on classifier scores) drastically change training data composition; failing to replicate these steps alters model capacity and final scores.
- Post-processing steps like quote replacement must be applied to model outputs before scoring to match official WMT evaluation scripts.
- BPE tokenizers are trained separately per language pair and domain; using a shared or mismatched tokenizer will skew n-gram alignment and BLEU computation.
Evidence (verbatim from paper)
resulting in a 43.8 BLEU score on Ru→En and 40.3 on En→Ru, surpassing prior year results.
Citation
@misc{subramanian2021nemo,
title={NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21},
author={Subramanian et al. (2021)},
year={2021},
note={arXiv:2111.08634}
}
- arXiv: 2111.08634