Multimed St Eval

multimed-st-eval

MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation — Le-Duc et al. (2025) (arXiv:2504.03546, 2025)

What this evaluates

Evaluates the capability of speech translation models to accurately convert medical speech across five languages (English, Vietnamese, German, French, Mandarin Chinese) into text. It probes both end-to-end and cascaded architectures, as well as the impact of multilingual vs. bilingual training and code-switching handling in a specialized medical domain.

Datasets

• MultiMed-ST — total 290000; splits: test (-1); repo https://github.com/leduckhai/MultiMed-ST

Metrics

• BLEU (primary) — range: [0, 100]
  • Standard n-gram overlap metric computing the geometric mean of modified precisions up to n=4, adjusted with a brevity penalty to penalize overly short translations. Values in the paper are scaled by 100.
• BERTScore (primary) — range: [0, 1]
  • Embedding-based metric that computes the F1 score between reference and hypothesis using contextual embeddings (e.g., BERT), matching tokens via cosine similarity.

Input / output format

Input: Audio recording of medical speech in a source language (en, vi, fr, de, or zh).

Output: Transcribed and translated text in the target language.

Scoring recipe

def compute_bleu(hypotheses, references):
    # Standard sentence-level BLEU with n=4 and brevity penalty, scaled to 100
    return 100 * nltk.translate.bleu_score.corpus_bleu(references, hypotheses)

def compute_bertscore(hypotheses, references):
    # BERTScore F1 using contextual embeddings
    scores = bertscore.score(hypotheses, references, lang='auto')
    return scores.f1.mean()

Common pitfalls

• Relying solely on n-gram metrics like BLEU may misrepresent semantic quality in medical translation; embedding-based metrics like BERTScore are recommended as complementary.
• Cascaded systems (ASR→MT) accumulate ASR errors that directly degrade translation quality, which end-to-end models may mitigate but often lack domain-specific tuning.
• Code-switching and low-resource language pairs (e.g., vi-de, zh-fr) show significantly lower performance, requiring careful data balancing or multi-task training.

Evidence (verbatim from paper)

Automatic MT metrics: To evaluate MT quality, two standard categories of evaluation metrics were utilized: n-gram overlap metrics (e.g., BLEU (Papineni et al. (2002)), TER (Snover et al. (2006)), METEOR (Banerjee and Lavie (2005)), ChrF (Popović (2015)), ROUGE (Lin (2004))) and embedding-based metrics (e.g., BERTScore (Zhang et al.)).

Citation

@misc{leduc2025multimedst,
  title={MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation},
  author={Le-Duc et al. (2025)},
  year={2025},
  note={arXiv:2504.03546}
}

• arXiv: 2504.03546