Spark Tts Eval

spark-tts-eval

Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens — Wang et al. (2025) (arXiv:2503.01710, 2025)

What this evaluates

Evaluates zero-shot text-to-speech generation by measuring speech intelligibility and speaker similarity across Chinese and English prompts. It also probes fine-grained control over voice attributes such as gender, pitch, and speaking rate.

Datasets

• Seed-TTS-eval — total ?; splits: test-zh (-1), test-en (-1); repo https://github.com/BytedanceSpeech/seed-tts-eval

Metrics

• CER/WER (primary) — range: percent
  • Character Error Rate (CER) for Chinese and Word Error Rate (WER) for English. Computed by transcribing the generated audio with an ASR model and comparing it to the reference text. Lower values indicate better intelligibility.
• SIM — range: [0, 1]
  • Speaker Similarity score based on cosine similarity between speaker embeddings extracted from the reference audio and the generated audio. Higher values indicate better voice cloning fidelity.

Input / output format

Input: Reference audio clip and a text prompt (for zero-shot TTS), or text prompt with attribute labels/values (for control tasks).

Output: Synthesized audio waveform matching the reference speaker and text content.

Scoring recipe

def evaluate(predictions, gold):
    cer_scores, wer_scores, sim_scores = [], [], []
    for pred_audio, ref_audio, ref_text in zip(predictions, gold):
        transcript = asr_model.transcribe(pred_audio)
        cer_scores.append(char_error_rate(ref_text, transcript))
        wer_scores.append(word_error_rate(ref_text, transcript))
        ref_emb = speaker_encoder.encode(ref_audio)
        pred_emb = speaker_encoder.encode(pred_audio)
        sim_scores.append(cosine_similarity(ref_emb, pred_emb))
    return {
        'CER': mean(cer_scores),
        'WER': mean(wer_scores),
        'SIM': mean(sim_scores)
    }

Common pitfalls

• Confusing CER (used for Chinese) with WER (used for English); the benchmark explicitly separates metrics by language.
• Using unofficial ASR or speaker embedding models, which drastically alters CER/WER and SIM scores compared to the official Seed-TTS-eval pipeline.
• Mixing training data or using non-standard splits, as the protocol strictly requires the official test-zh and test-en subsets.

Evidence (verbatim from paper)

To evaluate Spark-TTS’s zero-shot TTS capability, we assess its performance on Seed-TTS-eval and compare it with existing zero-shot TTS models. The results are presented in Table[4], where speech intelligibility is evaluated using the Character Error Rate (CER) for Chinese and the WER for English, following the Seed-TTS-eval. In terms of speaker similarity, while Spark-TTS is relatively weaker than multi-stage or NAR-based methods, it significantly outperforms the single-stage model Llasa.

Citation

@misc{wang2025sparktts,
  title={Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.01710}
}

• arXiv: 2503.01710