Thiomi Baseline Eval

thiomi-baseline-eval

The Thiomi Dataset: A Large-Scale Multimodal Corpus for Low-Resource African Languages — Mutisya et al. (2026) (arXiv:2603.29244, 2026)

What this evaluates

Evaluates the quality and utility of a multimodal corpus for low-resource African languages. It does so by training and testing baseline models for automatic speech recognition, machine translation, and text-to-speech across multiple languages.

Datasets

• Thiomi Dataset — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated as the sum of insertions, deletions, and substitutions divided by the total number of reference words, reported as a percentage.
• BLEU — range: percent
  • Standard n-gram precision metric with a brevity penalty to penalize overly short translations, reported on a 0–100 scale.
• MOS — range: [1, 5]
  • Mean Opinion Score derived from averaging 1–5 Likert scale ratings given by human evaluators for naturalness and intelligibility of synthesized speech.

Input / output format

Input: Audio recordings for ASR; parallel source-target text pairs for MT; raw text utterances for TTS.

Output: Predicted transcriptions for ASR; translated text for MT; synthesized audio waveforms for TTS.

Scoring recipe

# ASR: Word Error Rate
wer = (insertions + deletions + substitutions) / len(reference_words) * 100

# MT: BLEU (0-100 scale)
bleu = 100 * math.exp(sum(math.log(p_n) for p_n in ngram_precisions) / num_ngrams) * brevity_penalty

# TTS: Mean Opinion Score
mos = sum(rating for rating in evaluator_ratings) / len(evaluator_ratings)

Common pitfalls

• Tonal languages like Maasai and Kipsigis use complex phonology not fully captured by Latin orthography, which artificially inflates WER.
• BLEU scores are computed on a narrow, domain-constrained sentence set, so high scores do not generalize to open-domain translation.
• MOS evaluations rely on only 5 native speakers per language rating 50 utterances, which may not capture broader listener preferences or dialectal variation.

Evidence (verbatim from paper)

The Swahili result (3.24% WER on Common Voice) represents the best reported academic WER for Swahili ASR, surpassing the prior best system (XLS-R finetuned, 8.3%) by 61% relative improvement.

Citation

@misc{mutisya2026thiomi,
  title={The Thiomi Dataset: A Large-Scale Multimodal Corpus for Low-Resource African Languages},
  author={Mutisya et al. (2026)},
  year={2026},
  note={arXiv:2603.29244}
}

• arXiv: 2603.29244