Vietmed Asr Eval

vietmed-asr-eval

VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain — Le-Duc (2024) (arXiv:2404.05659, 2024)

What this evaluates

Evaluates automatic speech recognition (ASR) performance on Vietnamese medical domain audio. It measures how well models transcribe speech containing medical terminology and regional accents, assessing cross-domain transfer capabilities.

Datasets

• VietMed — total ?; splits: dev (-1); repo https://github.com/leduckhai/MultiMed

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference transcript.

Input / output format

Input: Audio recordings of Vietnamese speech in the medical domain, typically sampled at 8kHz or higher.

Output: Text transcript corresponding to the input audio.

Scoring recipe

def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• The best model epoch is selected based on WER on the development set, which may introduce validation set leakage if not properly separated from the final test set.
• The acoustic model uses generalized triphone states with 4501 labels, which can cause phoneme alignment issues for rare medical terms, artificially inflating WER.

Evidence (verbatim from paper)

We then chose the pre-training epoch to fine-tune with Framewise Cross-Entropy (fCE) loss that led to the best WERs on dev.

Citation

@misc{leduc2024vietmed,
  title={VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain},
  author={Le-Duc (2024)},
  year={2024},
  note={arXiv:2404.05659}
}

• arXiv: 2404.05659