Libritts Ssd Eval

libritts-ssd-eval

Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding — Lin et al. (2025) (arXiv:2505.15380, 2025)

What this evaluates

Evaluates the zero-shot speaker adaptation capability of an autoregressive speech synthesis model on unseen speakers. It measures content accuracy, voice cloning fidelity, and audio quality, while also quantifying inference speedup over standard autoregressive decoding.

Datasets

• LibriTTS — total 585; splits: train (-1), val (-1), test (500)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated by comparing the ground-truth transcript against the ASR output of the generated speech. Measures content consistency.
• SS — range: [0, 1]
  • Raw cosine similarity between speaker embedding vectors extracted by the ERes2Net model from the reference prompt and the generated audio.
• NMOS — range: [1, 5]
  • Objective metric for evaluating the perceptual quality of synthesized speech.
• LM-RTF — range: other
  • Real-Time Factor defined as the ratio of the time taken by the language model to generate the speech to the actual duration of the synthesized audio.

Input / output format

Input: Text transcript and a ~3-second reference audio prompt for zero-shot speaker cloning.

Output: Synthesized speech audio waveform (or discrete speech tokens).

Scoring recipe

def compute_metrics(generated_audio, prompt_audio, gold_text, generation_time, audio_duration):
    wer = whisper_large_v3.transcribe(generated_audio).word_error_rate(gold_text)
    ss = cosine_similarity(er2net_embed(prompt_audio), er2net_embed(generated_audio))
    lm_rtf = generation_time / audio_duration
    return {'WER': wer, 'SS': ss, 'LM-RTF': lm_rtf}

Common pitfalls

• Using an ASR model other than Whisper-large V3 for WER calculation will produce inconsistent results.
• Confusing objective NMOS with subjective MOS tests (SIM-MOS/NAT-MOS), which require separate human rater panels and 5-point scales.
• Evaluating on cross-speaker data without accounting for the zero-shot adaptation setup, as the test set specifically targets unseen speakers.

Evidence (verbatim from paper)

Referring to [5], we take word error rate (WER), speaker similarity (SS), and NMOS score as the objective metrics for a comprehensive evaluation of SSD. The Word Error Rate can measure content consistency. We use Whisper-large V3 as the Automatic Speech Recognition (ASR) model to calculate the WER. Regarding the Speaker Similarity, we employ the ERes2Net model to extract the speaker embedding vectors of the prompt speech and the generated one, and adopt their raw cosine similarity as the speaker similarity. The NMOS score is a widely-adopted metric for evaluating the objective quality of speech. We compute LM-RTF to assess the efficiency of the LM, which is defined as the ratio of the time taken by the LM to generate the speech to the duration of the synthesized speech.

Citation

@misc{lin2025accelerating,
  title={Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2505.15380}
}

• arXiv: 2505.15380