Accented Clinical Asr Eval

accented-clinical-asr-eval

Performant ASR Models for Medical Entities in Accented Speech — Afonja et al. (2024) (arXiv:2406.12387, 2024)

What this evaluates

Evaluates ASR models on African-accented clinical speech to measure how well they transcribe medical named entities (MNEs) like drug names, diagnoses, and lab results. It specifically probes the gap between standard word-level accuracy and clinically relevant entity recognition.

Datasets

• AfriSpeech — total ?; splits: train_general (21682), train_clinical (36318), train_both (58000), test_clinical (-1)

Metrics

• M-WER (primary) — range: [0, 1]
  • Standard Word Error Rate computed exclusively on tokens corresponding to medical named entities after fuzzy alignment between ASR output and ground truth.
• M-CER — range: [0, 1]
  • Standard Character Error Rate computed exclusively on characters of medical named entities after fuzzy alignment.
• medical NE Recall — range: [0, 1]
  • Proportion of ground-truth medical named entities correctly identified and aligned in the ASR output using fuzzy string matching.

Input / output format

Input: Audio recordings of African-accented clinical speech (general, clinical, or mixed domain).

Output: Raw transcribed text generated by the ASR model.

Scoring recipe

def compute_medical_metrics(asr_text, gold_text, entities_gold):
    # 1. Align ASR output with gold entities using MedTextAlign (fuzzy matching)
    aligned = medtextalign_align(asr_text, gold_text, entities_gold)
    # 2. Extract only tokens/chars belonging to aligned entities
    asr_ent = [t for t in asr_text if t in aligned]
    gold_ent = [t for t in gold_text if t in aligned]
    # 3. Compute standard WER/CER on entity subsets
    mwer = wer(asr_ent, gold_ent)
    mcer = cer(asr_ent, gold_ent)
    # 4. Compute NE Recall
    ne_recall = len(aligned) / len(entities_gold)
    return mwer, mcer, ne_recall

Common pitfalls

• Standard WER/CER metrics mask entity errors because ASR may correctly transcribe non-entity words while severely distorting medical terms.
• Exact string matching fails on ASR outputs due to phonetic variations in accented speech, necessitating fuzzy alignment to avoid undercounting entity errors.
• Fine-tuning on mixed domains can lead to overfitting or catastrophic forgetting of general speech patterns, requiring careful validation on the clinical test set.

Evidence (verbatim from paper)

revealing that despite low overall word error rates, medical named entity (MNE) recognition suffers from high error rates—particularly in drug names, diagnoses, and lab results—due to poor alignment between ASR outputs and ground truth entities. A novel fuzzy string matching algorithm (MedTextAlign) enables precise alignment of noisy ASR predictions with clinical entities, enabling the computation of medical NE Recall, medical WER (M-WER), and M-CER.

Citation

@misc{afonja2024performantasr,
  title={Performant ASR Models for Medical Entities in Accented Speech},
  author={Afonja et al. (2024)},
  year={2024},
  note={arXiv:2406.12387}
}

• arXiv: 2406.12387