Voxstream Tts Eval

voxstream-tts-eval

VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency — Torgashov et al. (2025) (arXiv:2509.15969, 2025)

What this evaluates

Evaluates a streaming text-to-speech model's ability to generate high-quality, speaker-similar, and intelligible audio from text prompts in both non-streaming and full-stream (word-by-word) settings. It probes zero-shot voice cloning, cross-sentence continuity, and real-time synthesis latency.

Datasets

• LibriSpeech test-clean — total ?; splits: test-clean (-1), long (-1)
• SEED-TTS test-en — total ?; splits: test-en (-1)

Metrics

• WER (%) — range: percent
  • Word Error Rate between the ASR transcription of synthesized speech and the input text. Whisper-large-v3 is used for SEED-TTS, while a HuBERT-based ASR is used for LibriSpeech.
• SPK-SIM — range: [0, 1]
  • Cosine similarity between WavLM-based ECAPA-TDNN embeddings extracted from the reference prompt and the generated audio.
• UTMOS — range: [0, 5]
  • MOS predictor score estimating audio quality.
• Naturalness (primary) — range: [0, 100]
  • MUSHRA-like user study rating on a 0–100 scale, computed as mean ± 95% CI.
• FPL (ms) — range: ms
  • First-packet latency, defined as the time to the first speech frame.
• RTF — range: other
  • Real-time factor, defined as the ratio of generated speech duration to wall-clock generation time.

Input / output format

Input: Text input (phonemes or raw text) provided sequentially word-by-word to simulate streaming LLM input, plus an audio prompt for speaker reference.

Output: Synthesized audio waveform (or acoustic tokens decoded to speech).

Scoring recipe

def compute_metrics(generated_audio, reference_audio, gold_text):
    # Intelligibility
    transcribed = asr_model(generated_audio)  # Whisper-large-v3 for SEED, HuBERT for LibriSpeech
    wer = word_error_rate(gold_text, transcribed)
    # Speaker Similarity
    ref_emb = speaker_encoder(reference_audio)  # WavLM-based ECAPA-TDNN
    gen_emb = speaker_encoder(generated_audio)
    spk_sim = cosine_similarity(ref_emb, gen_emb)
    # Quality
    utmos = utmos_predictor(generated_audio)
    return {'WER': wer, 'SPK-SIM': spk_sim, 'UTMOS': utmos}

Common pitfalls

• Different ASR backends are used for different datasets (Whisper-large-v3 for SEED-TTS, HuBERT for LibriSpeech), making cross-dataset WER comparisons invalid.
• Naturalness user studies must only include sentences where all baseline models achieve 0% WER to prevent intelligibility failures from biasing perceived naturalness scores.
• In full-stream evaluation, some baselines (e.g., CosyVoice2) re-synthesize the prompt when target text is shorter, artificially inflating WER; these cases were explicitly excluded by the authors.

Evidence (verbatim from paper)

We used three reproducible model-based metrics. For intelligibility, we report WER between the transcription of synthesized speech and the input text. For SEED-TTS test-en we used Whisper-large-v3 and followed metric calculation from the official SEED test repository. For LibriSpeech test-clean we used a HuBERT-based ASR and prepended the audio prompt to the generated continuation for WER, as in [15]. For speaker similarity, we computed cosine similarity (SPK-SIM) between embeddings from a WavLM-based ECAPA-TDNN for the prompt and synthesized speech. For quality, we used the UTMOS MOS predictor.

Citation

@misc{torgashov2025voxstream,
  title={VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency},
  author={Torgashov et al. (2025)},
  year={2025},
  note={arXiv:2509.15969}
}

• arXiv: 2509.15969