Wmt14 En Fr Eval

wmt14-en-fr-eval

Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — Cho et al. (2014) (arXiv:1406.1078, 2014)

What this evaluates

Evaluates the effectiveness of an RNN Encoder-Decoder architecture for statistical machine translation on English-to-French tasks. It measures how well the model learns phrase representations and improves translation quality over a traditional phrase-based baseline system.

Datasets

• WMT'14 English/French — total ?; splits: train (348000000), dev (-1), test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram precision metric for machine translation, computed as a geometric mean of unigram to 4-gram precisions with a brevity penalty. Scores are reported on tokenized French text.

Input / output format

Input: Source English sentence or phrase.

Output: Translated French sentence or phrase.

Scoring recipe

def compute_bleu(predictions, references):
    # predictions and references are lists of tokenized strings
    # Compute n-gram precisions for n=1..4
    # Apply brevity penalty based on length ratio
    # Return geometric mean of precisions * brevity penalty
    pass

Common pitfalls

• Vocabulary is strictly limited to the top 15,000 words, with all OOVs mapped to [UNK], which can artificially suppress BLEU scores for rare but correct translations.
• Development sets (newstest2012/2013) were used for both data selection and MERT tuning, creating a risk of dev-set overfitting before final evaluation on newstest2014.
• BLEU is calculated on French words after tokenization; inconsistent tokenization or punctuation handling compared to the Moses baseline will skew scores.

Evidence (verbatim from paper)

We evaluate our approach on the English/French translation task of the WMT'14 workshop. We used the test set newtest2012 and 2013 for data selection and weight tuning with MERT, and newtest2014 as our test set. The baseline phrase-based SMT system was built using Moses with default settings. This system achieves a BLEU score of 30.64 and 33.3 on the development and test sets, respectively (see Table 1).

Citation

@misc{cho2014learning,
  title={Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation},
  author={Cho et al. (2014)},
  year={2014},
  note={arXiv:1406.1078}
}

• arXiv: 1406.1078