Naijas2st Eval

naijas2st-eval

NaijaS2ST: A Multi-Accent Benchmark for Speech-to-Speech Translation in Low-Resource Nigerian Languages — Maltais et al. (2026) (arXiv:2604.16287, 2026)

What this evaluates

Evaluates speech-to-text and speech-to-speech translation capabilities across low-resource Nigerian languages (Hausa, Igbo, Yorùbá, Nigerian Pidgin) and English. It specifically probes how well cascaded, end-to-end, and AudioLLM architectures handle multi-accent variations and bidirectional translation directions.

Datasets

• NaijaS2ST — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• SSA-COMET (primary) — range: [0, 100]
  • Sentence-level semantic similarity metric based on COMET embeddings, adapted for speech translation. Scores higher for better semantic alignment between source and target, typically reported on a 0-100 scale.
• ChrF — range: [0, 100]
  • Character n-gram F-score measuring surface-level character overlap between reference and hypothesis. Highly sensitive to diacritics and spelling variations.
• SpBLEU — range: [0, 100]
  • Byte-level n-gram F-score for subword tokenization, used as an additional surface-form metric for translation quality.

Input / output format

Input: Source speech audio in Hausa, Igbo, Yorùbá, Nigerian Pidgin, or English, recorded with either Naija or British RP accent.

Output: Target text transcript (for S2TT) or target speech audio (for S2ST), depending on the pipeline configuration.

Scoring recipe

if task == 'S2TT':
    ssa_comet = compute_ssa_comet(hypothesis_text, reference_text)
    chrf = compute_chrf(hypothesis_text, reference_text)
elif task == 'S2ST':
    transcribed_hypothesis = asr_model.generate_transcript(hypothesis_speech)
    ssa_comet = compute_ssa_comet(transcribed_hypothesis, reference_text)
    chrf = compute_chrf(transcribed_hypothesis, reference_text)
# Note: For S2ST, the ASR model (Omnilingual-ASR 1B or Naija-Omni) must first transcribe the generated speech before metric computation.

Common pitfalls

• SSA-COMET can yield high semantic scores even when the model outputs the source language instead of the target language, due to its tolerance for lexical and surface-form mismatches.
• ChrF heavily penalizes languages with extensive diacritics (e.g., Yorùbá), making it an unreliable standalone metric for these languages and potentially masking genuine translation progress.
• Evaluation ASR models exhibit accent bias; using a base model like Omnilingual-ASR on Naija-accented speech artificially deflates S2ST scores unless the ASR is fine-tuned on the target accent (Naija-Omni).

Evidence (verbatim from paper)

For evaluation, we use Omnilingual-ASR 1B model for ASR on the speech outputs, then extract the SSA-COMET and ChrF scores from those transcripts. This discrepancy highlights a key limitation of embedding-based metrics: SSA-COMET is more tolerant to lexical and surface-form mismatches, whereas ChrF penalizes such deviations more directly.

Citation

@misc{maltais2026naijas2st,
  title={NaijaS2ST: A Multi-Accent Benchmark for Speech-to-Speech Translation in Low-Resource Nigerian Languages},
  author={Maltais et al. (2026)},
  year={2026},
  note={arXiv:2604.16287}
}

• arXiv: 2604.16287