Sagalee Asr Eval

sagalee-asr-eval

Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language — Turi Abu et al. (2025) (arXiv:2502.00421, 2025)

What this evaluates

Evaluates automatic speech recognition (ASR) performance on the Oromo language using real-world, crowd-sourced audio data. It measures how well different model architectures (Conformer trained from scratch, Whisper fine-tuned) transcribe spoken Oromo into text under varying acoustic conditions.

Datasets

• Sagalee — total 100; splits: train (94), dev (4), test (2); repo https://github.com/turinaf/sagalee

Metrics

• WER (primary) — range: percent
  • Word Error Rate (WER) is the standard ASR metric calculated as the minimum number of word edits (insertions, deletions, substitutions) required to transform the predicted transcription into the reference transcription, divided by the total number of words in the reference. Expressed as a percentage.

Input / output format

Input: Acoustic features extracted from Oromo speech audio recordings (x).

Output: Text transcription corresponding to the input audio (y).

Scoring recipe

def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    if total_words == 0: return 0.0
    edits = 0
    for pred, ref in zip(predictions, references):
        # Standard Levenshtein distance on word tokens
        dist = levenshtein_distance(pred.split(), ref.split())
        edits += dist
    return (edits / total_words) * 100

Common pitfalls

• The dataset size is reported in hours (93.6h/4.2h/2.4h) rather than sample counts, which can cause confusion when comparing to text-only benchmarks.
• Model evaluation uses an ensemble of checkpoints (average of 10 best for Conformer, average of last 3 for Whisper) rather than a single final checkpoint, which inflates reported performance compared to standard single-model baselines.
• WER is highly sensitive to tokenization; the paper uses BPE (nbpe=500) for training, so evaluation must use the exact same tokenizer to be reproducible.

Evidence (verbatim from paper)

The dataset is split into train, dev and test sets with 93.6hrs, 4.2hrs, and 2.4hrs size respectively. The performance of the trained models in terms of Word Error Rate (WER) is summarized in Table [IV]... Conformer AED and Conformer CTC models achieve WERs of 15.32% and 18.74%, respectively... The Whisper model achieves a significantly lower WER... 10.82%

Citation

@misc{turi2025sagalee,
  title={Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language},
  author={Turi Abu et al. (2025)},
  year={2025},
  note={arXiv:2502.00421}
}

• arXiv: 2502.00421