Covost2 St Mt Eval

covost2-st-mt-eval

CoVoST 2 and Massively Multilingual Speech-to-Text Translation — Wang et al. (2020) (arXiv:2007.10310, 2020)

What this evaluates

Evaluates multilingual speech-to-text translation and automatic speech recognition across 22 languages. It probes the model's ability to transcribe spoken audio and translate it into English (or from English) under monolingual, bilingual, and multilingual training regimes.

Datasets

• CoVoST 2 — total ?; splits: test (-1); repo https://github.com/facebookresearch/covost

Metrics

• BLEU (primary) — range: [0, 1]
  • Case-sensitive detokenized BLEU score computed using sacreBLEU with default options. For English-Chinese and English-Japanese directions, character-level BLEU is used instead of word-level.
• WER — range: percent
  • Word Error Rate calculated on tokenized, lowercased text with punctuation removed (except apostrophes and hyphens). Used for all languages except Japanese and Chinese.
• CER — range: percent
  • Character Error Rate used for Japanese and Chinese where word segmentation is not applied.

Input / output format

Input: 80-dimensional log mel-scale filter bank features (25ms window, 10ms shift, per-utterance CMVN) for ASR/ST; source text transcripts for MT/ST.

Output: Detokenized target text (translations for MT/ST, transcriptions for ASR).

Scoring recipe

if metric == 'BLEU':
    pred = detokenize(model_output)
    if lang in ['zh', 'ja']:
        score = sacrebleu.corpus_bleu([pred], [gold], tokenize='char')
    else:
        score = sacrebleu.corpus_bleu([pred], [gold])
elif metric in ['WER', 'CER']:
    pred = preprocess(pred) # sacreBLEU tokenizer, lowercase, remove punctuation except apostrophes/hyphens
    gold = preprocess(gold)
    if lang in ['ja', 'zh']:
        score = compute_cer(pred, gold)
    else:
        score = compute_wer(pred, gold)

Common pitfalls

• English-Chinese and English-Japanese directions require character-level BLEU instead of standard word-level BLEU.
• WER/CER calculation mandates specific preprocessing: tokenization via sacreBLEU tokenizers, lowercasing, and removal of punctuation (except apostrophes and hyphens).
• Model checkpoint selection differs by task: MT uses the single best validation checkpoint, while ASR/ST averages the last 5 checkpoints.

Evidence (verbatim from paper)

For MT and ST, we report case-sensitive detokenized BLEU using sacreBLEU with default options, except for English-Chinese and English-Japanese where we report character-level BLEU. For ASR, we report character error rate (CER) on Japanese and Chinese (no word segmentation) and word error rate (WER) on the other languages using VizSeq. Before calculating WER (CER), sentences are tokenized by sacreBLEU tokenizers, lowercased and with punctuation removed (except for apostrophes and hyphens).

Citation

@misc{wang2020covost2,
  title={CoVoST 2 and Massively Multilingual Speech-to-Text Translation},
  author={Wang et al. (2020)},
  year={2020},
  note={arXiv:2007.10310}
}

• arXiv: 2007.10310