United Medasr Asr Eval

united-medasr-asr-eval

High-precision medical speech recognition through synthetic data and semantic correction: UNITED-MEDASR — Banerjee et al. (2024) (arXiv:2412.00055, 2024)

What this evaluates

This evaluation measures the transcription accuracy of a fine-tuned automatic speech recognition model across four diverse speech benchmarks. It specifically probes the model's robustness to different speaking styles, accents, and linguistic contexts after applying a noise reduction step and a BART-based semantic correction pipeline.

Datasets

• LibriSpeech — total ?; splits: test-clean (-1)
• Europarl-ASR — total ?; splits: EN Guest-test (-1)
• TED-LIUM — total ?; splits: test (-1)
• FLEURS — total ?; splits: en (-1)

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • Standard ASR metric calculated as (S + D + I) / N, where S is substitutions, D is deletions, I is insertions, and N is the number of words in the reference. The paper explicitly normalizes both reference and hypothesis by lowercasing, collapsing multiple spaces to a single space, and splitting into word lists before computation.

Input / output format

Input: Audio recordings (speech data) from benchmark datasets, optionally pre-processed with a noise reduction algorithm.

Output: Transcribed text string corresponding to the input audio.

Scoring recipe

def calculate_wer(hypothesis, reference):
    # Text normalization per paper protocol
    hyp = hypothesis.lower().split()
    ref = reference.lower().split()
    # Standard Levenshtein-based WER calculation
    dist = levenshtein_distance(hyp, ref)
    n = len(ref)
    if n == 0: return 0.0
    return (dist / n) * 100  # Returns percentage

Common pitfalls

• Failing to apply the exact text normalization pipeline (lowercasing, collapsing spaces) before WER calculation, which can artificially inflate error rates.
• Calculating WER on the raw ASR output instead of the final BART-corrected output, as the protocol explicitly evaluates the full pipeline.
• Using the wrong test splits (e.g., LibriSpeech test-other instead of test-clean) or ignoring the specific Europarl-ASR EN Guest-test split, leading to non-comparable results.

Evidence (verbatim from paper)

These datasets enable researchers to compute important metrics such as the Word Error Rate (WER), which reflects the accuracy of the ASR system by measuring the differences between the generated and actual transcriptions. We applied a series of text transformations to both the actual and transcribed text. First, all the text was converted to lowercase to ensure consistency. Then, we removed any extra spaces and replaced them with a single space to standardise spacing. Following that, any sequences of multiple spaces were reduced to just one. Finally, the text was split into lists of words based on spaces, making it easier to compare the actual and transcribed versions.

Citation

@misc{banerjee2024unitedmedasr,
  title={High-precision medical speech recognition through synthetic data and semantic correction: UNITED-MEDASR},
  author={Banerjee et al. (2024)},
  year={2024},
  note={arXiv:2412.00055}
}

• arXiv: 2412.00055