Opus Mt Eval

opus-mt-eval

Democratizing Neural Machine Translation with OPUS-MT — Tiedemann et al. (2022) (arXiv:2212.01936, 2022)

What this evaluates

Evaluates the translation quality of OPUS-MT models across diverse language pairs using standard automatic metrics and specialized linguistic test suites. It probes general-purpose translation capability, lexical ambiguity disambiguation, and cross-lingual robustness.

Datasets

• Flores — total ?; splits: test (-1)
• Tatoeba — total ?; splits: test (-1)
• MuCoW — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: [0, 100] percent
  • Standard n-gram overlap metric between predicted and reference translations. The paper also reports spBLEU (sentence-piece tokenized), chrF/chrf++ (character n-gram F-score), and COMET (cross-lingual neural metric trained on human judgments).
• spBLEU — range: [0, 100] percent
  • Sentence-piece tokenized BLEU score, used for more consistent cross-lingual comparison.
• chrF++ — range: [0, 100] percent
  • Character n-gram F-score, noted as more reliable across languages than BLEU.
• COMET — range: [0, 1]
  • Cross-lingual neural evaluation metric trained on human judgments to better correlate with human quality assessments.

Input / output format

Input: Source sentence in the source language

Output: Target sentence in the target language

Scoring recipe

def evaluate(predictions, references, metric='BLEU'):
    scores = []
    for pred, ref in zip(predictions, references):
        if metric == 'BLEU':
            scores.append(calculate_bleu(pred, ref))
        elif metric == 'spBLEU':
            scores.append(calculate_spbleu(pred, ref))
        elif metric == 'chrF++':
            scores.append(calculate_chrf(pred, ref))
        elif metric == 'COMET':
            scores.append(comet_model.predict(pred, ref))
    return sum(scores) / len(scores) if scores else 0

Common pitfalls

• BLEU scores are problematic to compare across languages; chrF is preferred for reliability.
• Models are general-purpose and not directly comparable to WMT shared task systems due to different training data and domain tuning.
• MuCoW only evaluates lexical ambiguity for nouns and does not cover broader linguistic phenomena or fluency.

Evidence (verbatim from paper)

Currently, we support BLEU [33], spBLEU [32], chrF [34], chrf++ [35] and COMET [36] scores but other measures may be added once they become available from our systematic test procedures.

Citation

@misc{tiedemann2022opusmt,
  title={Democratizing Neural Machine Translation with OPUS-MT},
  author={Tiedemann et al. (2022)},
  year={2022},
  note={arXiv:2212.01936}
}

• arXiv: 2212.01936