Swivuriso Asr Eval

swivuriso-asr-eval

Swivuriso: The South African Next Voices Multilingual Speech Dataset — Marivate et al. (2025) (arXiv:2512.02201, 2025)

What this evaluates

Evaluates automatic speech recognition (ASR) capabilities across seven South African languages. It measures how well pre-trained speech models can transcribe spontaneous and scripted audio in low-resource, domain-specific contexts (agriculture, healthcare, general).

Datasets

• Swivuriso — total 3000; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the number of substitutions, deletions, and insertions required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference. Lower is better.
• CER — range: percent
  • Character Error Rate: identical to WER but computed at the character level instead of word level. Lower is better.

Input / output format

Input: Raw audio recordings (speech) in seven South African languages, spanning scripted and unscripted domains.

Output: Predicted text transcript corresponding to the input audio.

Scoring recipe

def compute_wer(predictions, references):
    # Align predicted and reference word sequences
    # Count substitutions (S), deletions (D), insertions (I)
    # N = number of words in reference
    wer = (S + D + I) / N
    return wer * 100  # reported as percent

Common pitfalls

• Zero-shot WER can exceed 100% for low-resource languages due to high insertion/deletion rates, which may be misinterpreted as model failure rather than a known ASR artifact.
• Cross-dataset comparisons require careful downsampling to equal durations (~6 hours), which can introduce sampling bias if not stratified by domain or speaker.
• Training is reported in optimization steps rather than epochs; comparing step counts across models with different learning rates or batch sizes requires normalization.

Evidence (verbatim from paper)

Table[6] summarises the Word Error Rate (WER) achieved by each model at early (1,000 steps) and final (10,000 steps) training checkpoints. Continued fine-tuning yields substantial improvements across all models, with Whisper-large-v3-Turbo achieving the largest absolute reduction of 0.22, followed by Wav2Vec-BERT at 0.27 and MMS at 0.15.

Citation

@misc{marivate2025swivuriso,
  title={Swivuriso: The South African Next Voices Multilingual Speech Dataset},
  author={Marivate et al. (2025)},
  year={2025},
  note={arXiv:2512.02201}
}

• arXiv: 2512.02201