Oversmoothing Rate Eval

oversmoothing-rate-eval

Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling — Kulikov et al. (2021) (arXiv:2112.08914, 2021)

What this evaluates

This evaluation probes the tendency of autoregressive neural machine translation models to prematurely terminate sequences by assigning high probability to short prefixes. It measures how well a model balances sequence length distribution and translation quality under beam search decoding.

Datasets

• IWSLT'17 — total ?; splits: train (-1), val (800), test (8000)
• WMT'16 En->De — total ?; splits: train (4500000), val (3000), test (3000)
• WMT'19 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• oversmoothing_rate (primary) — range: other
  • Averages the log-probability and normalized rank of the <eos> token across all positions $t$ in generated sequences, then averages across all translations. Higher values indicate a stronger tendency to prematurely end sequences.
• BLEU — range: percent
  • Standard n-gram overlap metric computed via sacreBLEU on detokenized predictions and references.

Input / output format

Input: Source sentence in the source language (e.g., German, French, Chinese, Russian).

Output: Detokenized target language translation sequence generated via beam search decoding.

Scoring recipe

# Compute BLEU
bleu_score = sacrebleu.corpus_bleu(predictions, references)

# Compute oversmoothing_rate
eos_log_probs = []
for pred in predictions:
    for t in range(1, len(pred)):
        eos_log_probs.append(model.log_prob('<eos>', prefix=pred[:t]))
oversmoothing_rate = sum(eos_log_probs) / len(eos_log_probs)

Common pitfalls

• The evaluation explicitly disables length normalization and length penalty during decoding to isolate oversmoothing effects.
• Results are highly sensitive to beam search size; metrics must be reported across multiple beam configurations.
• Early stopping is triggered by the training objective function on the validation set, not by translation quality metrics.

Evidence (verbatim from paper)

We compute and report BLEU scores using sacreBLEU on detokenized predictions. We vary beam sizes to study their effect in-depth. We set the lower- and upper-bound of a generated translation to be, respectively, 0 and $1.2\cdot l_{x}+10$, where $l_{x}$ is the length of the source $x$. We do not use either length normalization nor length penalty, in order to study the impact of oversmoothing on decoding faithfully.

Citation

@misc{kulikov2021oversmoothing,
  title={Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling},
  author={Kulikov et al. (2021)},
  year={2021},
  note={arXiv:2112.08914}
}

• arXiv: 2112.08914