Asr Clinical Continual Eval

asr-clinical-continual-eval

Navigating the Reality Gap: Privacy-Preserving On-Device Continual Adaptation of ASR for Clinical Telephony — Chauhan et al. (2025) (arXiv:2512.16401, 2025)

What this evaluates

This evaluation probes an ASR model's ability to continuously adapt to noisy, rural clinical telephony speech while retaining its baseline performance on standard general-domain speech. It specifically measures the trade-off between target-domain transcription accuracy and catastrophic forgetting of pre-trained linguistic knowledge.

Datasets

• Gram Vaani — total ?; splits: test (-1)
• Kathbath — total ?; splits: val (3151), train_replay (25800)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated as (Substitutions + Deletions + Insertions) / Total Words in reference. Measures transcription accuracy at the word level.
• CER — range: percent
  • Character Error Rate calculated as (Substitutions + Deletions + Insertions) / Total Characters in reference. Provides finer-grained phonetic accuracy analysis.

Input / output format

Input: Raw audio recordings (8kHz telephonic speech upsampled to 16kHz for clinical data; standard read speech for general domain).

Output: Text transcription corresponding to the input audio.

Scoring recipe

def compute_wer(predictions, references):
    S, D, I = edit_distance(predictions, references)
    N = sum(len(ref.split()) for ref in references)
    return (S + D + I) / N

Common pitfalls

• Prior work reported results on a 5-hour Development set; this benchmark strictly uses the 3-hour Evaluation set (GV_Eval_3h) for fair comparison.
• The evaluation simulates a continual learning stream by processing 103 hours of training data sequentially, not as a single batch, which significantly impacts adaptation dynamics.
• CER is reported for phonetic/dialectal analysis but WER is the primary headline metric for overall transcription accuracy.

Evidence (verbatim from paper)

We evaluate performance using two standard metrics: Word Error Rate (WER) and Character Error Rate (CER). WER measures transcription accuracy at the word level, while CER provides a finer-grained analysis of phonetic accuracy, particularly useful for agglutinative languages and dialectal variations.

Citation

@misc{chauhan2025navigating,
  title={Navigating the Reality Gap: Privacy-Preserving On-Device Continual Adaptation of ASR for Clinical Telephony},
  author={Chauhan et al. (2025)},
  year={2025},
  note={arXiv:2512.16401}
}

• arXiv: 2512.16401