qhjqhj00 clip-tts-eval
Clip-TTS: Contrastive Text-content and Mel-spectrogram, A High-Quality Text-to-Speech Method based on Contextual Semantic Understanding — Liu (2025) (arXiv:2502.18889, 2025)
What this evaluates
Evaluates the naturalness and quality of synthesized speech across single-speaker, multi-speaker, and multi-emotion TTS models. It measures how closely generated audio matches human ground truth in terms of overall speech quality and emotional similarity.
Datasets
- Baker — total 10000; splits: test (-1)
- AISHELL3 — total ?; splits: test (-1)
- LJSpeech — total 13100; splits: test (-1)
- LibriTTS — total ?; splits: test (-1)
- Emotional Speech Dataset (ESD) — total ?; splits: test (-1)
Metrics
MOS(primary) — range: [1, 5]- Mean Opinion Score: human listeners rate the overall naturalness and quality of synthesized speech on a standard 1–5 scale. Scores are averaged across listeners with 95% confidence intervals reported.
CMOS— range: [-3, 3]- Comparative Mean Opinion Score: listeners compare synthesized speech to a reference/original speech on a 6-level scale from -3 to +3. Positive scores indicate the synthesized speech is better/more similar, negative scores indicate it is worse/less similar.
Input / output format
Input: Text transcription (and optional emotion/style conditioning) provided to the TTS model.
Output: Mel-spectrograms generated by the model, subsequently converted to waveform audio via a pre-trained HiFi-GAN vocoder.
Scoring recipe
def compute_mos(predictions, gold):
ratings = [r for r in predictions if r is not None]
return sum(ratings) / len(ratings) if ratings else 0.0
def compute_cmos(predictions, gold):
ratings = [r for r in predictions if r is not None]
return sum(ratings) / len(ratings) if ratings else 0.0
Common pitfalls
- MOS scores are heavily influenced by the inherent recording quality and speaker characteristics of the source dataset, not just the model's architecture.
- CMOS measures emotional similarity to the original speech rather than absolute synthesis quality, so negative values do not necessarily mean poor audio fidelity.
- Human evaluation protocols vary widely in listener count, instructions, and confidence interval calculation, making cross-paper comparisons difficult without standardized guidelines.
Evidence (verbatim from paper)
The speech quality evaluation method I used is the MOS [37] score. ... On the emotional speech dataset, I used CMOS scores to evaluate the differences between the speech generated by Clip-TTS for different emotions and the original speech. The CMOS scores are divided into 6 levels, ranging from -3 to 3. A score between -1 and -3 indicates that the quality of the synthesized speech is progressively worse than the original speech, while a score between +1 and +3 indicates the opposite.
Citation
@misc{liu2025cliptts,
title={Clip-TTS: Contrastive Text-content and Mel-spectrogram, A High-Quality Text-to-Speech Method based on Contextual Semantic Understanding},
author={Liu (2025)},
year={2025},
note={arXiv:2502.18889}
}
- arXiv: 2502.18889