Un Corpus Translation Eval

un-corpus-translation-eval

Zero-Shot Dual Machine Translation — Sestorain et al. (2018) (arXiv:1805.10338, 2018)

What this evaluates

Evaluates zero-shot and supervised machine translation quality across multiple language pairs using a multilingual encoder-decoder architecture. It probes the model's ability to translate between unseen language pairs (e.g., Spanish-French) using only monolingual data and reinforcement learning, without parallel training data for the target pair.

Datasets

• United Nations Parallel Corpus (UN corpus) — total ?; splits: test (4000), dev (4000)

Metrics

• BLEU (primary) — range: percent
  • Standard sentence-level BLEU score computed over the test set. It measures n-gram precision (up to 4-grams) with a brevity penalty to penalize overly short translations.

Input / output format

Input: Source language sentence tokenized via shared 32K BPE vocabulary.

Output: Target language sentence tokenized via shared 32K BPE vocabulary.

Scoring recipe

def compute_bleu(predictions, references):
    # predictions, references: list of lists of BPE tokens
    bleu_scores = []
    for pred, ref in zip(predictions, references):
        # Standard sentence-level BLEU with 4-gram precision and brevity penalty
        bleu_scores.append(sentence_bleu([ref], pred, smoothing_method=SmoothingFunction().method1))
    return sum(bleu_scores) / len(bleu_scores) * 100

Common pitfalls

• BLEU is evaluated strictly on the official 2015 UN test/dev sets (4k sentences each), not on the full parallel corpus or monolingual data.
• Zero-shot directions (e.g., es→fr) are scored without any parallel training data for that pair, relying on shared embeddings and RL rewards.
• BPE tokenization (32K merges) is shared across languages, meaning BLEU is computed on subword units rather than raw words.

Evidence (verbatim from paper)

The corpus provides official development and test sets composed of the documents released in 2015. Both sets comprise 4,000 sentences, aligned for all the six languages. Table 1: BLEU scores on the UN corpus test set. Each line reports the BLEU scores of the corresponding translation direction.

Citation

@misc{sestorain2018zeroshotdual,
  title={Zero-Shot Dual Machine Translation},
  author={Sestorain et al. (2018)},
  year={2018},
  note={arXiv:1805.10338}
}

• arXiv: 1805.10338