En Ca Biomedical Translation Eval

en-ca-biomedical-translation-eval

English-Catalan Neural Machine Translation in the Biomedical Domain through the cascade approach — Costa-jussà et al. (2018) (arXiv:1803.07139, 2018)

What this evaluates

Evaluates English-to-Catalan machine translation quality in the biomedical domain using a two-stage cascade pivot strategy (English→Spanish→Catalan) versus direct translation, measuring lexical overlap and fluency via BLEU scores on domain-specific test sets.

Datasets

• WMT Biomedical test set — total ?; splits: test (-1)
• El Periódico test set — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram precision metric with brevity penalty. Calculated case-sensitively using the t2t-bleu script from tensor2tensor, equivalent to mteval-v14.pl from Moses.

Input / output format

Input: Source sentence in English (for EN-ES and EN-CA) or Spanish (for ES-CA).

Output: Target sentence in Spanish (for EN-ES) or Catalan (for ES-CA and EN-CA cascade).

Scoring recipe

def compute_bleu(predictions, references):
    # predictions and references are lists of raw strings
    # Case-sensitive tokenization is applied
    bleu = t2t_bleu(predictions, references) # equivalent to mteval-v14.pl
    return bleu * 100  # reported as percentage (e.g., 41.38)

Common pitfalls

• BLEU is computed case-sensitively, which differs from standard case-insensitive MT evaluation protocols.
• The cascade system's input for the second stage is machine-translated Spanish, meaning errors compound across stages and are not isolated to a single model.
• The test set for the cascade is referred to as the 'translated WMT Biometrical test set', requiring careful alignment of the English source with the Catalan reference despite the intermediate pivot.

Evidence (verbatim from paper)

The English-to-Spanish translation obtains a BLEU score of 46.55 in the test set of the WMT Biomedical test set while the Spanish-to-Catalan translation obtains a BLEU score of 86.89 in the El Periódico test set. The cascaded translation achieves a BLEU score of 41.38 in the translated WMT Biometrical test set. All BLEU scores are case-sensitive and where obtained with script t2t-bleu from the tensor2tensor framework, whose results are equivalent to those from mteval-v14.pl from the Moses package.

Citation

@misc{costa2018englishcatalan,
  title={English-Catalan Neural Machine Translation in the Biomedical Domain through the cascade approach},
  author={Costa-jussà et al. (2018)},
  year={2018},
  note={arXiv:1803.07139}
}

• arXiv: 1803.07139