Grad Tts Eval

grad-tts-eval

Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech — Popov et al. (2021) (arXiv:2105.06337, 2021)

What this evaluates

Evaluates text-to-speech synthesis quality, inference efficiency, and probabilistic modeling accuracy of a diffusion-based model. It probes the trade-off between synthesis fidelity and computational cost by varying reverse diffusion steps, and measures human-perceived audio quality against strong baselines.

Datasets

• LJSpeech — total ?; splits: train (-1), test (500)

Metrics

• MOS (primary) — range: [1, 5]
  • Average of 10 human ratings per audio sample on a 9-point Likert scale (1.0 to 5.0 in 0.5 increments).
• Log-likelihood — range: other
  • Average data log-likelihood estimated over 50 test sentences using the instantaneous change of variables formula and Hutchinson’s trace estimator.
• RTF — range: other
  • Real-Time Factor: ratio of total inference time to total generated audio duration.

Input / output format

Input: Phonemized text sequence

Output: Synthesized audio waveform

Scoring recipe

def compute_mos(predictions, gold):
    ratings = [human_listen_test(audio) for audio in predictions]
    return mean(ratings)

def compute_rtf(predictions, gold):
    total_time = sum(inference_time(audio) for audio in predictions)
    total_dur = sum(audio_duration(audio) for audio in predictions)
    return total_time / total_dur

def compute_log_likelihood(predictions, gold):
    lls = [compute_log_likelihood_score(text) for text in gold[:50]]
    return mean(lls)

Common pitfalls

• Diffusion loss convergence does not guarantee high synthesis quality; inaccurate gradient predictions for a small subset of time steps can severely degrade output despite low loss.
• Baseline comparisons may be skewed if unofficial implementations are used (e.g., FastSpeech in this work).
• Inference speed vs. quality trade-off is highly sensitive to the number of reverse diffusion steps; N=10 achieves near-optimal MOS, making higher N values computationally wasteful.

Evidence (verbatim from paper)

For Mean Opinion Score (MOS) estimation we synthesized 40 sentences from the test set with each model. The assessors were asked to estimate the quality of synthesized speech on a nine-point Likert scale, the lowest and the highest scores being 1 point (“Bad”) and 5 points (“Excellent”) with a step of 0.5 point. To ensure the reliability of the obtained results, only Master assessors were assigned to complete the listening test. Each audio was evaluated by 10 assessors.

Citation

@misc{popov2021gradtts,
  title={Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech},
  author={Popov et al. (2021)},
  year={2021},
  note={arXiv:2105.06337}
}

• arXiv: 2105.06337