Eurospeech Asr Eval

eurospeech-asr-eval

EuroSpeech: A Multilingual Speech Corpus — Pfisterer et al. (2025) (arXiv:2510.00514, 2025)

What this evaluates

Assesses the utility of the EuroSpeech multilingual corpus for fine-tuning automatic speech recognition (ASR) models. It measures the reduction in word error rate achieved by training on this corpus compared to baseline models across under-resourced European languages.

Datasets

• EuroSpeech — total 61000; splits: train (-1), test (-1)

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • The ratio of edit operations (substitutions, deletions, insertions) required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference. Calculated as (S + D + I) / N.

Input / output format

Input: Audio segments paired with reference human transcripts. For alignment training, noisy ASR transcriptions are also provided.

Output: Predicted text transcripts generated by the fine-tuned ASR model.

Scoring recipe

def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        total_errors += levenshtein_distance(pred.split(), ref.split())
        total_words += len(ref.split())
    return total_errors / total_words if total_words > 0 else 0.0

Common pitfalls

• Baseline WER varies drastically across languages, so absolute WER values are less informative than relative reduction percentages.
• The evaluation conflates alignment pipeline quality with ASR model capacity, as improvements depend heavily on the initial model's multilingual capabilities.

Evidence (verbatim from paper)

Language selection was motivated by two factors: (1) these six languages exhibited the highest baseline WER with Whisper v3 Turbo, allowing demonstration of meaningful improvements with limited computational resources, and (2) poor baseline ASR performance creates additional challenges for our alignment pipeline, as ASR transcriptions for these languages contain more errors, providing a rigorous test of our pipeline’s ability to match noisy ASR outputs to the correct segments in human transcripts.

Citation

@misc{pfisterer2025eurospeech,
  title={EuroSpeech: A Multilingual Speech Corpus},
  author={Pfisterer et al. (2025)},
  year={2025},
  note={arXiv:2510.00514}
}

• arXiv: 2510.00514