Mosel Maltese Asr Eval

mosel-maltese-asr-eval

MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages — Gaido et al. (2024) (arXiv:2410.01036, 2024)

What this evaluates

Evaluates automatic speech recognition (ASR) performance on low-resource Maltese speech data. It measures the accuracy of a sequence-to-sequence model in transcribing audio into text after training on filtered open-source speech corpora.

Datasets

• VoxPopuli (Maltese subset) — total 9000; splits: train (-1); repo https://github.com/hlt-mt/mosel

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • Computed as (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=reference word count. Lower values indicate better transcription accuracy.

Input / output format

Input: Audio segments represented as 80 Mel-filterbank features extracted every 10 ms with a 25 ms window.

Output: Text transcripts generated as a sequence of tokens from an 8,000-size SentencePiece vocabulary.

Scoring recipe

def compute_wer(predictions, references):
    # Normalize predictions and references using Whisper Normalizer
    norm_preds = whisper_normalizer(predictions)
    norm_refs = whisper_normalizer(references)
    # Compute Word Error Rate using JiWER
    wer = jiwer.wer(norm_refs, norm_preds)
    return wer

Common pitfalls

• ASR models frequently generate textual hallucinations, such as repetitive spurious words or long noisy strings, especially when background noise or music is present in the audio.
• Metadata language tags may be inconsistent with actual spoken content, leading to misaligned training data if not filtered via automatic Language Identification (LID).
• WER computation requires careful normalization; raw Whisper outputs often contain formatting artifacts that must be stripped before evaluation to avoid inflated error rates.

Evidence (verbatim from paper)

Results in Word Error Rate (WER) are computed using the Whisper Normalizer and, then, JiWER for computing the metric.

Citation

@misc{gaido2024mosel,
  title={MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages},
  author={Gaido et al. (2024)},
  year={2024},
  note={arXiv:2410.01036}
}

• arXiv: 2410.01036