Wmt21 Biomedical Translation Eval

wmt21-biomedical-translation-eval

Summer: WeChat Neural Machine Translation Systems for the WMT22 Biomedical Translation Task — Li et al. (2022) (arXiv:2211.15022, 2022)

What this evaluates

Evaluates Chinese-to-English machine translation performance on biomedical texts. It measures how well a neural MT system can translate domain-specific terminology and syntax while handling case, punctuation, and subword tokenization conventions.

Datasets

• WMT21 OK-aligned biomedical test set — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard sentence-level BLEU score computed over n-gram precisions (typically up to 4-grams) with a brevity penalty. Scores are reported as percentages multiplied by 100.

Input / output format

Input: Chinese source sentences, pre-segmented by an in-house tool, normalized for punctuation, with case markers (UU, U) prepended to words, and tokenized via BPE (32K operations).

Output: English target sentences, tokenized via BPE (32K operations), post-processed with Moses detokenizer.perl to restore spacing and punctuation.

Scoring recipe

def compute_bleu(predictions, references):
    # predictions and references are lists of detokenized strings
    # Standard BLEU-4 with brevity penalty
    return nltk.translate.bleu_score.corpus_bleu(
        [[ref] for ref in references],
        predictions,
        smoothing_function=nltk.translate.bleu_score.SmoothingFunction().method4
    ) * 100

Common pitfalls

• Case handling requires special tokens (UU, U) before scoring; raw lowercase conversion will distort BLEU.
• Detokenization must be applied to English outputs using Moses scripts before BLEU computation to match the paper's protocol.
• BPE tokenization (32K vocab) is applied to both source and target; scoring should be done on detokenized text, not raw tokens.

Evidence (verbatim from paper)

The experimental results of Chinese → English on WMT21 OK-aligned biomedical test set are shown in Table 2. Compared with the baseline model (Baseline_TAG), the in-domain bilingual data (+IND-BIO) provided by WMT22 shared biomedical translation task brings a huge improvement, with 6.5 point increase in BLEU score.

Citation

@misc{li2022summer,
  title={Summer: WeChat Neural Machine Translation Systems for the WMT22 Biomedical Translation Task},
  author={Li et al. (2022)},
  year={2022},
  note={arXiv:2211.15022}
}

• arXiv: 2211.15022