Speakstream Streaming Tts Eval

speakstream-streaming-tts-eval

SpeakStream: Streaming Text-to-Speech with Interleaved Data — Richard He Bai et al. (2025) (arXiv:2505.19206, 2025)

What this evaluates

Evaluates the quality and latency of streaming text-to-speech models that generate audio incrementally from interleaved text and speech inputs. It probes the model's ability to maintain speech accuracy and naturalness while minimizing first-token latency under streaming constraints.

Datasets

• LJSpeech — total ?; splits: dev (-1)
• LibriSpeech — total ?; splits: dev-clean (-1)

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • Calculated by transcribing the generated speech using WhisperX (base.en) and comparing the output text to the ground truth transcript. Standard word-level error rate counting substitutions, insertions, and deletions.
• TTS Latency — range: ms
  • Time elapsed between the TTS model receiving its first word and generating its first frame (or first actual phoneme). Measured in milliseconds.
• Human Naturalness/Coherence — range: [1, 5]
  • Average rating on a 1 to 5 scale provided by human annotators for each segment's naturalness and coherence.

Input / output format

Input: Streaming text tokens (words) provided sequentially, optionally interleaved with previously generated speech features or audio segments.

Output: Streaming Mel spectrogram features (dMel) or 24kHz audio waveform chunks.

Scoring recipe

def evaluate_wer(generated_audio, ground_truth_text):
    transcribed = whisperx_base_en.transcribe(generated_audio)
    return calculate_wer(transcribed, ground_truth_text)

def evaluate_latency(model, first_word):
    t_start = time.time()
    model.process(first_word)
    _ = model.get_first_audio_chunk()
    t_end = time.time()
    return (t_end - t_start) * 1000

Common pitfalls

• Applying non-streaming TTS models directly to streaming segments causes severe hallucination and WER > 200%.
• Measuring latency from the first audio byte rather than the first actual phoneme overestimates real-world response time.
• Assuming larger context windows (m) always improve accuracy; performance drops beyond m=5 due to text repetition artifacts.

Evidence (verbatim from paper)

For TTS evaluation, we utilize WhisperX (“base.en”) to transcribe our generated speech into text and calculate the Word Error Rate (WER). ... We report three latency metrics: 1. Total latency: Time elapsed between the TTS model receiving its first word and the audio player outputting the first waveform chunk. 2. Vocoder latency: Time elapsed between the vocoder receiving its first frame input and generating the first chunk of waveform. 3. TTS latency: Time elapsed between the TTS model receiving its first word and generating its first frame.

Citation

@misc{bai2025speakstream,
  title={SpeakStream: Streaming Text-to-Speech with Interleaved Data},
  author={Richard He Bai et al. (2025)},
  year={2025},
  note={arXiv:2505.19206}
}

• arXiv: 2505.19206