Wmt19 Slt Eval

wmt19-slt-eval

UDS--DFKI Submission to the WMT2019 Similar Language Translation Shared Task — Pal et al. (2019) (arXiv:1908.06138, 2019)

What this evaluates

Evaluates machine translation quality between similar languages (Czech to Polish) using a multi-encoder transformer trained on out-of-domain data filtered by cross-entropy differences. Probes the model's ability to adapt to low-resource similar language pairs via domain adaptation and data selection.

Datasets

• WMT19 SLT Shared Task dataset — total ?; splits: dev (-1), test (-1)

Metrics

• BLEU (primary) — range: percent
  • Computes modified n-gram precision across multiple n-gram orders, combined with a brevity penalty to penalize overly short translations. Standard WMT implementation uses corpus-level BLEU-4.
• TER — range: percent
  • Translation Edit Rate measures the minimum number of edits (insertions, deletions, substitutions, and shifts) required to transform the hypothesis into the reference, normalized by reference length.

Input / output format

Input: Source Czech sentence processed with word-level information on the first encoder and BPE tokens on the second encoder.

Output: Target Polish sentence generated as a sequence of BPE tokens.

Scoring recipe

def compute_metrics(hypotheses, references):
    # BLEU (corpus-level, standard WMT implementation)
    bleu_score = sacrebleu.corpus_bleu(hypotheses, [references]).score
    # TER (minimum edit distance normalized by reference length)
    ter_score = tercom.compute(hypotheses, references).score
    return {'bleu': bleu_score, 'ter': ter_score}

Common pitfalls

• The development set is explicitly used as the test set for internal evaluation before the official submission deadline.
• Final predictions are generated by averaging the last 8 best training checkpoints rather than selecting a single best checkpoint.
• BPE vocabulary is learned jointly on both languages, which affects tokenization and OOV handling compared to standard monolingual BPE.

Evidence (verbatim from paper)

We evaluate our approach with development data which is used as test case before submission. We use BLEU (Papineni et al., 2002) and TER (Snover et al., 2006).

Citation

@misc{pal2019udsdfki,
  title={UDS--DFKI Submission to the WMT2019 Similar Language Translation Shared Task},
  author={Pal et al. (2019)},
  year={2019},
  note={arXiv:1908.06138}
}

• arXiv: 1908.06138