Kunkado Nyana Eval Eval

kunkado-nyana-eval-eval

Kunnafonidilaw ka Cadeau: an ASR dataset of present-day Bambara — Diarra et al. (2025) (arXiv:2512.19400, 2025)

What this evaluates

Evaluates automatic speech recognition (ASR) models on spontaneous, radio-based Bambara speech containing real-world artifacts like code-switching, overlapping speakers, and background noise. It measures transcription accuracy under pragmatic normalization conditions.

Datasets

• Kunkado Test — total ?; splits: test (-1)
• Nyana-Eval — total ?; splits: test (-1)

Metrics

• WER (%) (primary) — range: percent
  • Word Error Rate: percentage of words incorrectly transcribed. Calculated as (Substitutions + Deletions + Insertions) / Total Words in reference.
• CER (%) — range: percent
  • Character Error Rate: percentage of characters incorrectly transcribed. Calculated as (Substitutions + Deletions + Insertions) / Total Characters in reference.

Input / output format

Input: Raw audio recordings of spontaneous Bambara speech (radio broadcasts).

Output: Normalized text transcription. Acoustic event tags, code-switching markers, punctuation, and diacritics are removed from both reference and prediction before scoring.

Scoring recipe

def score_asr(predictions, references):
    # Apply normalization: remove tags, code-switching markers, punctuation, diacritics
    pred_clean = normalize_text(predictions)
    ref_clean = normalize_text(references)
    # Calculate edit distances
    wer = edit_distance(pred_clean, ref_clean) / len(ref_clean.split())
    cer = edit_distance(pred_clean, ref_clean) / len(ref_clean)
    return wer * 100, cer * 100

Common pitfalls

• Failing to strip acoustic event tags and code-switching markers from both reference and hypothesis before scoring, which artificially inflates error rates.
• Applying training-time normalizations (e.g., removing numbers/diacritics) inconsistently during evaluation, leading to non-comparable WER/CER values.
• Confusing the 33.47-hour human-reviewed training subset with the 5-hour test set used for reporting.

Evidence (verbatim from paper)

We evaluated all the models on a 5 hour test set taken from the Kunkado data, and Nyana-Eval, a small, stratified human evaluation dataset with only 45 entries of 3 minutes total duration... We apply the same normalization steps as explained in section 3 and remove the tags from both the reference and the prediction before calculating the WER and CER.

Citation

@misc{diarra2025kunnafonidilaw,
  title={Kunnafonidilaw ka Cadeau: an ASR dataset of present-day Bambara},
  author={Diarra et al. (2025)},
  year={2025},
  note={arXiv:2512.19400}
}

• arXiv: 2512.19400