Chren Bleu Eval

chren-bleu-eval

ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization — Zhang et al. (2020) (arXiv:2010.04791, 2020)

What this evaluates

Evaluates machine translation quality between Cherokee and English, focusing on low-resource, morphologically complex translation. It probes both in-domain and out-of-domain generalization, as well as the reliability of automatic metrics versus human judgment for polysynthetic languages.

Datasets

• ChrEn — total 14000; splits: dev (-1), test (-1), out-dev (-1), out-test (-1); repo https://github.com/ZhangShiyue/ChrEn

Metrics

• BLEU (primary) — range: percent
  • Detokenized and case-sensitive BLEU score computed using SacreBLEU. Standard n-gram precision with brevity penalty, applied to raw token sequences without lowercasing.

Input / output format

Input: Source sentence in Cherokee or English.

Output: Target sentence in English or Cherokee.

Scoring recipe

import sacrebleu
# predictions and references are lists of detokenized strings
score = sacrebleu.corpus_bleu(predictions, [references], lowercase=False, tokenize='none')
return score.score

Common pitfalls

• BLEU scores may not correlate with human preference for Cherokee due to its polysynthetic morphology and complex word formation.
• Out-of-domain performance trends differ significantly from in-domain; SMT often outperforms NMT on out-of-domain data despite NMT being better in-domain.

Evidence (verbatim from paper)

As shown in Table 5, human preference does not always follow the trends of BLEU scores. For English-Cherokee translation, though the RNN-NMT+BERT (N5) has a better BLEU score than SMT+BT (S3) (12.2 vs. 9.9), it is liked less by humans (21 vs. 29), indicating that BLEU is possibly not a suitable for Cherokee evaluation.

Citation

@misc{zhang2020chren,
  title={ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization},
  author={Zhang et al. (2020)},
  year={2020},
  note={arXiv:2010.04791}
}

• arXiv: 2010.04791