Voxtream2 Eval

voxtream2-eval

VoXtream2: Full-stream TTS with dynamic speaking rate control — Torgashov et al. (2026) (arXiv:2603.13518, 2026)

What this evaluates

This evaluation probes a full-stream text-to-speech model's ability to generate intelligible, natural-sounding speech while dynamically controlling the speaking rate in real-time. It measures objective intelligibility, speaker similarity, audio quality, generation latency, and the accuracy of speaking-rate control against target rates.

Datasets

• Emilia speaking-rate dataset — total ?; splits: test (-1)

Metrics

• WER (%) (primary) — range: percent
  • Word Error Rate calculated as the percentage of incorrectly transcribed words relative to the ground truth text.
• SPK-SIM — range: [0, 1]
  • Speaker Similarity score computed as the cosine similarity between speaker embeddings of the generated audio and the reference acoustic prompt.
• UTMOS — range: other
  • Universal TTS Mean Opinion Score, an objective predictor of perceived audio quality and naturalness.
• FPL (ms) — range: ms
  • First Packet Latency, measuring the time elapsed from input submission to the generation of the first audio packet.
• RTF — range: [0, 1]
  • Real-Time Factor, calculated as the ratio of total generated audio duration to the total inference time.
• Corr. — range: [0, 1]
  • Pearson correlation coefficient between the target speaking rate (SPS) control signal and the actual synthesized speaking rate extracted via phoneme alignment.

Input / output format

Input: Text input, acoustic prompt (reference speech), and target speaking rate or control signal (e.g., SPS value or linearly varying signal).

Output: Synthesized audio waveform.

Scoring recipe

def compute_metrics(audio, text, prompt_audio, target_sps, control_signal):
    wer = wer_engine(transcribe(audio), text)
    spk_sim = cosine_similarity(speaker_encoder(audio), speaker_encoder(prompt_audio))
    utmos = utmos_model(audio)
    fpl = time_to_first_packet()
    rtf = len(audio) / inference_time()
    target_rates = extract_sps(control_signal)
    gen_rates = extract_sps_from_alignments(audio)
    corr = pearsonr(target_rates, gen_rates)
    return {'WER': wer, 'SPK-SIM': spk_sim, 'UTMOS': utmos, 'FPL': fpl, 'RTF': rtf, 'Corr.': corr}

Common pitfalls

• WER can be artificially inflated at slow speaking rates due to model-generated fillers and word repetitions, which sound natural but penalize transcription accuracy.
• SPK-SIM and UTMOS are not reported for the full-stream setting because they showed no significant difference compared to non-streaming evaluation.
• Spark-TTS is excluded from speaker similarity (SPK-SIM) evaluation because its controllable version does not support voice cloning.
• UTMOS ratings are sensitive to evaluation design and may reflect overall audio quality or background noise rather than strictly naturalness.

Evidence (verbatim from paper)

We benchmark the performance of our model against prior work in Table 4. VoXtream2 achieves the lowest FPL and RTF among publicly available full-stream models. We also evaluate different GPU inference optimizations. While torch.compile provides the best runtime performance, it requires longer graph compilation time compared to CUDA Graphs.

Citation

@misc{torgashov2026voxstream2,
  title={VoXtream2: Full-stream TTS with dynamic speaking rate control},
  author={Torgashov et al. (2026)},
  year={2026},
  note={arXiv:2603.13518}
}

• arXiv: 2603.13518