Multimed Asr Eval

multimed-asr-eval

MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder — Khai Le-Duc et al. (2024) (arXiv:2409.14074, 2024)

What this evaluates

Evaluates multilingual automatic speech recognition (ASR) performance on medical domain audio across five languages. It probes the model's ability to accurately transcribe spoken medical terminology under diverse recording conditions, accents, and speaking roles.

Datasets

• MultiMed — total ?; splits: test (-1); repo https://github.com/leduckhai/MultiMed

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the minimum number of word edits (insertions, deletions, substitutions) required to transform the predicted transcript into the ground truth, normalized by the total number of words in the reference.
• CER — range: percent
  • Character Error Rate: the minimum number of character edits (insertions, deletions, substitutions) required to transform the predicted transcript into the ground truth, normalized by the total number of characters in the reference.

Input / output format

Input: Audio recordings of medical speech in five languages (Vietnamese, English, German, French, Mandarin) with varying recording conditions, accents, and speaking roles.

Output: Text transcript of the spoken audio.

Scoring recipe

def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edits = levenshtein_distance(normalize(pred), normalize(ref))
        total_errors += edits
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• ASR metrics are highly sensitive to text normalization (e.g., handling of numbers, punctuation, case, and language-specific tokenization like spaces in Mandarin vs. Vietnamese).
• The paper evaluates both WER and CER but does not specify the exact normalization or alignment algorithm used, which can cause significant variance across different evaluation toolkits.

Evidence (verbatim from paper)

To assess the performance of the ASR models, we employed two standard evaluation metrics: Word Error Rate (WER) and Character Error Rate (CER).

Citation

@misc{leduc2024multimed,
  title={MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder},
  author={Khai Le-Duc et al. (2024)},
  year={2024},
  note={arXiv:2409.14074}
}

• arXiv: 2409.14074