Polish Asr Eval

polish-asr-eval

Quality of Automatic Speech Recognition -- Polish Language case study -- from Wav2Vec to Scribe ElevenLabs — Pietroń et al. (2026) (arXiv:2603.02246, 2026)

What this evaluates

Evaluates the transcription accuracy of various automatic speech recognition (ASR) models on Polish-language audio, contrasting read-speech benchmarks with spontaneous, noisy medical consultations to probe domain generalization.

Datasets

• Mozilla Common Voice (MCV) Polish — total ?; splits: train (-1), dev (-1), test (-1)
• Multilingual LibriSpeech (MLS) Polish — total ?; splits: train (-1), dev (-1), test (-1)
• Medical interview corpus — total ?; splits: test (-1)

Metrics

• Word Error Rate (WER) (primary) — range: [0, 1] | percent
  • Standard ASR metric computed as the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcription into the reference text, divided by the total number of words in the reference.

Input / output format

Input: Audio recordings (MP3 or WAV) of Polish speech, including read audiobooks, conversational interviews, and medical consultations.

Output: Plain text transcription of the spoken audio.

Scoring recipe

def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        ref_words = ref.split()
        pred_words = pred.split()
        total_words += len(ref_words)
        total_errors += edit_distance(ref_words, pred_words)
    return total_errors / total_words if total_words > 0 else 0.0

Common pitfalls

• Standard benchmarks use read speech and clean audio, failing to capture the acoustic variability, overlapping speech, and medical terminology of real-world clinical settings.
• Model performance on general benchmarks does not reliably predict domain-specific accuracy, as shown by the superior performance of ElevenLabs Scribe in medical contexts despite Whisper's strong benchmark results.

Evidence (verbatim from paper)

The recordings are used for transcription, and the reference texts are used for model evaluation (for the experiments, Polish parts from multilingual open-source databases were selected - Mozilla Common Voice (MCV), Multilingual LibriSpeech (MLS)): ... Each sample contains an audio file and its manually verified transcript.

Citation

@misc{pietron2026quality,
  title={Quality of Automatic Speech Recognition -- Polish Language case study -- from Wav2Vec to Scribe ElevenLabs},
  author={Pietroń et al. (2026)},
  year={2026},
  note={arXiv:2603.02246}
}

• arXiv: 2603.02246