Nmt Kd Eval

nmt-kd-eval

Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation — Zhang et al. (2023) (arXiv:2305.08096, 2023)

What this evaluates

Evaluates neural machine translation quality under knowledge distillation settings. It measures how well student models can replicate teacher performance on standard cross-lingual translation benchmarks.

Datasets

• WMT'14 En-De — total ?; splits: test (-1)
• WMT'14 En-Fr — total ?; splits: test (-1)
• WMT'16 En-Ro — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric between system output and reference translations, typically computed with tokenized text.
• COMET — range: [0, 1]
  • A reference-based neural metric that uses a pre-trained model to score translation quality based on semantic similarity and fluency.

Input / output format

Input: English source sentence tokenized with BPE (32k merges).

Output: Target language sentence tokenized with the shared BPE vocabulary.

Scoring recipe

bleu = sacrebleu.corpus_bleu(predictions, [references])
comet_model = load_comet_model('unbabel/wmt22-comet-da')
scores = comet_model.predict({'src': sources, 'mt': predictions, 'ref': references})
comet_score = scores['scores'].mean()

Common pitfalls

• BLEU scores are highly sensitive to tokenization; the paper uses BPE with 32k merges, so standard sacrebleu tokenization may differ.
• COMET scores depend on the specific model version; the paper cites Rei et al. (2020) but does not specify the exact checkpoint used.
• Knowledge distillation experiments often report only test BLEU; the paper also reports COMET, which requires a separate inference step and model loading.

Evidence (verbatim from paper)

To make the results more convincing, we report both BLEU and COMET Rei et al. ([2020]) scores in Tab.[6] Using Transformerbig as the teacher, our method can boost the Transformerbase students by +1.04/+0.60/+1.11 BLEU scores and +4.52/+2.57/+4.80 COMET scores on three tasks, respectively.

Citation

@misc{zhang2023towards,
  title={Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2305.08096}
}

• arXiv: 2305.08096