Habibi Tts Eval

habibi-tts-eval

Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis — Chen et al. (2026) (arXiv:2601.13802, 2026)

What this evaluates

Evaluates zero-shot and unified-dialectal text-to-speech synthesis across multiple Arabic dialects. It measures transcription accuracy, speaker similarity, and audio naturalness to assess how well a model preserves dialectal features and voice identity without dialect-specific fine-tuning.

Datasets

• Habibi Benchmark — total ?; splits: test (-1)

Metrics

• WER-O (primary) — range: percent
  • Word Error Rate computed by the Omnilingual-ASR-LLM-7B model on generated speech compared to ground truth text. Lower is better.
• WER-S — range: percent
  • Word Error Rate computed by dialect-specific ASR models on generated speech compared to ground truth text. Lower is better.
• SIM — range: [0, 1]
  • Speaker similarity score computed using the WavLM speaker verification model between the generated audio and the reference audio. Higher is better.
• UTMOS — range: other
  • Naturalness score predicted by the UTMOS model on the generated audio. Higher is better.

Input / output format

Input: Arabic text (dialect-specific) and a reference audio clip (for zero-shot/in-context voice cloning).

Output: Synthesized speech audio waveform.

Scoring recipe

# WER-O/S: Transcribe generated audio with ASR, compare to GT text
asr = load_asr("Omnilingual-ASR-LLM-7B" or dialect_specific)
pred = asr.transcribe(generated_audio)
wer = compute_wer(pred, ground_truth_text)

# SIM: Speaker similarity via WavLM
wavlm = load_wavlm()
sim = wavlm.compute_similarity(generated_audio, reference_audio)

# UTMOS: Naturalness score
utmos = load_utmos()
utmos_score = utmos.predict(generated_audio)

Common pitfalls

• Multilingual ASR models may exhibit cross-dialect recognition bias, incorrectly 'correcting' dialectal speech, which is why the authors report both WER-O and WER-S.
• Ground truth samples from ASR corpora often contain higher noise levels than TTS outputs, which can artificially inflate WER scores for the reference speech.
• Commercial baseline comparison requires careful selection and LLM-validated reference audio to ensure authentic dialect characteristics.

Evidence (verbatim from paper)

We measure three conventional metrics: word error rate (WER) using ASR models, speaker similarity (SIM) leveraging the speaker verification model WavLM Chen et al. ([2022]), and naturalness with UTMOS Saeki et al. ([2022]). Notably, we report two sets of WER scores: (1) WER-O, evaluated with Omnilingual-ASR-LLM-7B model Omnilingual et al. ([2025]) (v1 with a fixed batch size of 64); and (2) WER-S, derived from dialect-specific ASR models, most of which trained following VietASR Zhuo et al. ([2025]) except for EGY and MAR (two XLSR fine-tuned models on Hugging Face666IbrahimAmin/egyptian-arabic-wav2vec2-xlsr-53,777boumehdi/wav2vec2-large-xlsr-moroccan-darija are employed). The rationale for introducing both is to enable a more reliable conclusion, given that a multilingual model risks cross-dialect recognition bias (e.g., incorrectly “correcting” speech that lacks dialect features); conversely, specialized models often suffer from poor generalization and noise resistance.

Citation

@misc{chen2026habibi,
  title={Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2601.13802}
}

• arXiv: 2601.13802