Back Translation Wake Sleep Eval

back-translation-wake-sleep-eval

Explaining and Generalizing Back-Translation through Wake-Sleep — Cotterell et al. (2018) (arXiv:1806.04402, 2018)

What this evaluates

Evaluates neural machine translation models on English-German, German-English, English-Latvian, and Latvian-English translation tasks. It probes the effectiveness of iterative back-translation (wake-sleep extension) compared to standard back-translation and baseline MLE training across supervised and semi-supervised domain adaptation scenarios.

Datasets

• WMT 2017 — total ?; splits: train (-1), dev (-1), test (-1)
• TED (IWSLT 2014) — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• BLEU (SACREBLEU v1.2.3) (primary) — range: percent
  • Standard n-gram overlap metric. Evaluated using SACREBLEU v1.2.3 with specific tokenization and smoothing: TED uses case.lc+numrefs.1+smooth.exp+tok.13a.version.1.2.3, WMT uses case.mixed+numrefs.1+smooth.exp+tok.13a.version.1.2.3.

Input / output format

Input: Source sentence in English, German, or Latvian.

Output: Target sentence in the corresponding target language.

Scoring recipe

import sacrebleu

def compute_bleu(hypotheses, references, dataset_type):
    if dataset_type == 'TED':
        score = sacrebleu.corpus_bleu(
            hypotheses, references,
            lowercase=True,
            smooth_method='exp',
            tokenize='13a'
        )
    elif dataset_type == 'WMT':
        score = sacrebleu.corpus_bleu(
            hypotheses, references,
            lowercase=False,
            smooth_method='exp',
            tokenize='13a',
            case_mixed=True
        )
    return score.score

Common pitfalls

• Using different tokenization or smoothing settings than SACREBLEU v1.2.3, which drastically changes BLEU scores.
• Confusing the baseline (Iteration 0, MLE-only) with the back-translation iterations when reporting improvements.
• Failing to use the correct test set sizes or mixing up dev/test splits for early stopping.

Evidence (verbatim from paper)

The models are evaluated with respect to BLEU (Papineni et al., 2002) using the SACREBLEU tool (v.1.2.3) (Post, 2018) on tokenized (WMT: recased) system outputs.

Citation

@misc{cotterell2018explaining,
  title={Explaining and Generalizing Back-Translation through Wake-Sleep},
  author={Cotterell et al. (2018)},
  year={2018},
  note={arXiv:1806.04402}
}

• arXiv: 1806.04402