Dysarthric Asr Eval

dysarthric-asr-eval

Bridging ASR and LLMs for Dysarthric Speech Recognition: Benchmarking Self-Supervised and Generative Approaches — Ahmed Aboeitta et al. (2025) (arXiv:2508.08027, 2025)

What this evaluates

Evaluates the ability of ASR and LLM-enhanced decoding models to accurately transcribe dysarthric speech across varying severity levels and domains. It probes robustness to phonetic distortions, grammatical consistency, and cross-dataset generalization.

Datasets

• TORGO — total ?; splits: test (-1)
• UASpeech — total ?; splits: test (-1)

Metrics

• WER (primary) — range: [0, 1]
  • Word Error Rate, calculated as the normalized edit distance (substitutions, deletions, insertions) between the predicted and reference transcriptions, divided by the number of words in the reference.
• CER — range: [0, 1]
  • Character Error Rate, defined as CER = (S + D + I) / N, where S, D, and I are the number of substitutions, deletions, and insertions, and N is the number of characters in the reference transcription.

Input / output format

Input: Dysarthric speech audio recordings.

Output: Text transcription of the spoken utterance.

Scoring recipe

def compute_metrics(predictions, references):
    wer_vals, cer_vals = [], []
    for pred, ref in zip(predictions, references):
        pred_words = pred.split()
        ref_words = ref.split()
        wer_vals.append(edit_distance(pred_words, ref_words) / len(ref_words) if ref_words else 0.0)
        cer_vals.append(edit_distance(list(pred), list(ref)) / len(ref) if ref else 0.0)
    return sum(wer_vals) / len(wer_vals), sum(cer_vals) / len(cer_vals)

Common pitfalls

• Relying solely on WER may overlook semantic fluency improvements captured by qualitative analysis and CER.
• Cross-dataset generalization performance degrades significantly, so in-dataset results alone overestimate real-world robustness.

Evidence (verbatim from paper)

To assess transcription quality beyond WER, we analyze Character Error Rate (CER) and sample transcriptions, offering a finer evaluation of phoneme distortions and semantic accuracy.

Citation

@misc{aboeitta2025bridging,
  title={Bridging ASR and LLMs for Dysarthric Speech Recognition: Benchmarking Self-Supervised and Generative Approaches},
  author={Ahmed Aboeitta et al. (2025)},
  year={2025},
  note={arXiv:2508.08027}
}

• arXiv: 2508.08027