Cuni Wmt22 Csuk Eval

cuni-wmt22-csuk-eval

CUNI Systems for the WMT22 Czech-Ukrainian Translation Task — Popel et al. (2022) (arXiv:2212.00486, 2022)

What this evaluates

Evaluates machine translation quality for Czech-Ukrainian and Ukrainian-Czech translation using constrained back-translation systems and a proprietary unconstrained system. Probes the impact of data preprocessing techniques like romanization and ensemble methods on translation performance.

Datasets

• Flores 101 development set — total ?; splits: dev (-1)
• WMT22 Czech-Ukrainian test set — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric for machine translation, typically reported as a percentage (0-100).
• COMET — range: [0, 1]
  • Reference-based neural metric that predicts translation quality using a pre-trained model.
• chrF — range: percent
  • Character-level n-gram F-score measuring overlap between reference and hypothesis at the character level.

Input / output format

Input: Source sentence in Czech or Ukrainian.

Output: Translated sentence in the target language (Ukrainian or Czech).

Scoring recipe

def score_mt(hypotheses, references, metric='bleu'):
    if metric == 'bleu':
        return sacrebleu.corpus_bleu(hypotheses, [references]).score
    elif metric == 'comet':
        return comet_model.predict(hypotheses, references)
    elif metric == 'chrf':
        return sacrebleu.corpus_chrf(hypotheses, [references]).score

Common pitfalls

• COMET scores are unreliable for Czech-Ukrainian as the language pair was not in COMET's training data.
• Small differences in automatic metrics (BLEU, chrF, COMET) should not be overinterpreted as significant.
• Batch size during training had a larger impact on results than the proposed preprocessing techniques.

Evidence (verbatim from paper)

In this section, we report BLEU scores on the Flores 101 development set that we used to make our decisions about the system development and the final automatic scores. Note that the validation set is very different from the test set. The validation set consists of clean and rather complicated sentences from Wikipedia articles, whereas the WMT 22 test set is noisy user-generated text from the logs of the production deployment of Charles Translator.[5] Automatic evaluation on the WMT22 test set is presented in Table 5. Both the constrained systems and Charles Translator show comparable results. The tagged BT system reaches a slightly higher COMET score than the Block BT system, however, Czech-Ukrainian was not in the training data of the COMET score, which make the score unreliable for this particular language pair.

Citation

@misc{popel2022cunisystems,
  title={CUNI Systems for the WMT22 Czech-Ukrainian Translation Task},
  author={Popel et al. (2022)},
  year={2022},
  note={arXiv:2212.00486}
}

• arXiv: 2212.00486