Niletts Eval

niletts-eval

LLM-to-Speech: A Synthetic Data Pipeline for Training Dialectal Text-to-Speech Models — Khamis et al. (2026) (arXiv:2602.15675, 2026)

What this evaluates

Evaluates the quality of a fine-tuned Text-to-Speech model for Egyptian Arabic dialect synthesis. It measures speech intelligibility, acoustic fidelity, and speaker similarity compared to a baseline model.

Datasets

• NileTTS — total ?; splits: test (-1); repo https://github.com/KickItLikeShika/NileTTS

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • Computed by synthesizing speech from text, transcribing it with Whisper Large, and calculating the word-level edit distance against the original text. Lower values indicate higher intelligibility.
• Character Error Rate (CER) — range: percent
  • A finer-grained intelligibility metric computed at the character level using the same Whisper transcription and original text comparison.
• Speaker Similarity — range: [-1, 1]
  • Cosine similarity between ECAPA-TDNN speaker embeddings extracted from the synthesized audio and the original reference audio. Higher values indicate better voice cloning.
• Evaluation Loss — range: other
  • Combined text and mel-spectrogram cross-entropy loss as defined by the XTTS architecture.

Input / output format

Input: Egyptian Arabic text prompts and reference speaker audio for conditioning.

Output: Synthesized audio waveform corresponding to the input text.

Scoring recipe

def compute_metrics(text, ref_audio, tts_model):
    syn_audio = tts_model.generate(text, ref_audio)
    transcribed = whisper_large.transcribe(syn_audio)
    wer = compute_wer(transcribed, text)
    cer = compute_cer(transcribed, text)
    ref_emb = ecapa_tdnn(ref_audio)
    syn_emb = ecapa_tdnn(syn_audio)
    spk_sim = cosine_similarity(ref_emb, syn_emb)
    return {'WER': wer, 'CER': cer, 'Spk_Sim': spk_sim}

Common pitfalls

• WER and CER rely on Whisper Large for transcription, which may introduce its own biases or errors when processing Egyptian Arabic dialectal speech.
• Speaker similarity uses ECAPA-TDNN embeddings trained on standard/clean speech, potentially underestimating voice fidelity for dialectal variations.
• Checkpoint selection is automated based on metric stabilization (~34k steps), but manual listening validation was only performed on 50 randomly sampled utterances.

Evidence (verbatim from paper)

We use the following evaluation metrics, computed on the held-out evaluation set: Evaluation Loss: Combined text and mel-spectrogram cross-entropy loss as defined by the XTTS architecture. Word Error Rate (WER): We synthesize speech from text, transcribe it using Whisper Large Radford et al. ([2022]), and compute WER against the original text. Lower WER indicates higher intelligibility. Character Error Rate (CER): A finer-grained intelligibility metric computed at the character level. Speaker Similarity: Cosine similarity between ECAPA-TDNN Desplanques et al. ([2020]) speaker embeddings of synthesized and reference audio. Higher similarity indicates better voice cloning.

Citation

@misc{khamis2026niletts,
  title={LLM-to-Speech: A Synthetic Data Pipeline for Training Dialectal Text-to-Speech Models},
  author={Khamis et al. (2026)},
  year={2026},
  note={arXiv:2602.15675}
}

• arXiv: 2602.15675