Syncspeech Eval

syncspeech-eval

SyncSpeech: Low-Latency and Efficient Dual-Stream Text-to-Speech based on Temporal Masked Transformer — Sheng et al. (2025) (arXiv:2502.11094, 2025)

What this evaluates

Evaluates a dual-stream text-to-speech model's ability to generate high-quality, speaker-similar speech with low latency and high efficiency under streaming and offline conditions. It probes the model's robustness to complex text, alignment accuracy, and real-time generation speed compared to autoregressive and interleaved baselines.

Datasets

• LibriSpeech test-clean — total ?; splits: test (-1)
• SeedTTS test-zh — total 2000; splits: test (2000)
• SeedTTS test-hard — total 400; splits: test (400)

Metrics

• WER — range: percent
  • Word Error Rate calculated by transcribing generated speech with Whisper-V3 (English) or Paraformer (Mandarin) and comparing against original text transcripts. Lower is better.
• SS — range: percent
  • Speaker Similarity measured using an ERes2Net-based speaker verification model. Higher percentage indicates better voice preservation.
• FPL-A — range: other
  • First-packet latency assuming text is already available. Measured as the time required for the model to generate enough speech tokens to fill the decoder chunk size (15).
• FPL-L — range: other
  • First-packet latency accounting for upstream LLM text generation. Includes time to receive streaming text tokens before speech generation begins.
• RTF (primary) — range: percent
  • Real-Time Factor calculated as the ratio of the total duration of generated speech to the total time taken by the model. Reported as a percentage in the paper.
• MOS-N — range: other
  • Mean Opinion Score for speech naturalness. 10 native listeners score 100 selected sentences on a 1-to-5 scale.

Input / output format

Input: Text input provided either as a complete sequence or in a streaming token-by-token fashion.

Output: Generated speech audio tokens, decoded into waveform via an off-the-shelf streaming speech decoder.

Scoring recipe

def compute_metrics(generated_audio, original_text, reference_audio, model_time, speech_duration):
    # WER
    asr_transcript = asr_model(generated_audio)
    wer = 100 * edit_distance(asr_transcript, original_text) / len(original_text)
    # SS
    ss = 100 * speaker_verification_model(generated_audio, reference_audio)
    # RTF
    rtf = 100 * (speech_duration / model_time)
    # MOS-N
    mos_n = mean(listener_scores_1_to_5)
    return {'WER': wer, 'SS': ss, 'RTF': rtf, 'MOS-N': mos_n}

Common pitfalls

• FPL-A and FPL-L measure different latency scenarios; FPL-L includes upstream LLM generation time, so comparing them directly without noting the scenario is misleading.
• RTF is reported as a percentage (e.g., 0.07%) in the paper, whereas the standard convention is a ratio (e.g., 0.0007). Users must multiply by 100 to match the table values.
• MOS-N is subjective and depends on the listener pool; scores from different papers cannot be directly compared without identical evaluation protocols.

Evidence (verbatim from paper)

For speech robustness, we chose Whisper-V3 and Paraformer as the ASR models for English and Mandarin, respectively, to transcribe the generated speech. Then, we calculated the WER compared to the original transcriptions to evaluate the spech robustness. We adopted the ERes2Net-based speaker verification model to evaluate speaker similarity (SS). We selected 100 sentences from each system and invited 10 native listeners to conduct a subjective MOS evaluation for speech naturalness (MOS-N), scoring from 1 to 5. In terms of latency and efficiency, we compared the performance of various models on a single A800 GPU. Due to the off-the-shelf speech decoder, we evaluate the latency and efficiency of the text-to-token stage across all models, except for F5-TTS. We calculated the time required for the number of speech tokens to reach the chunk size of the speech decoder as First-packet latency (FPL). There are two scenarios: one assumes the text is already available (FPL-A), while the other involves receiving output from the upstream LLM model (FPL-L), accounting for the time required for text generation. For the real-time factor (RTF), we measure the ratio of the total duration of generate

Citation

@misc{sheng2025syncspeech,
  title={SyncSpeech: Low-Latency and Efficient Dual-Stream Text-to-Speech based on Temporal Masked Transformer},
  author={Sheng et al. (2025)},
  year={2025},
  note={arXiv:2502.11094}
}

• arXiv: 2502.11094