Glm Tts Eval

glm-tts-eval

GLM-TTS Technical Report — Cui et al. (2025) (arXiv:2512.14291, 2025)

What this evaluates

Evaluates text-to-speech systems on pronunciation accuracy, speaker similarity, and emotional expressiveness across standard Chinese/English benchmarks and challenging internal datasets. It also assesses vocoder quality using objective and subjective audio metrics to measure overall synthesis fidelity.

Datasets

• Seed-TTS-eval — total ?; splits: test-zh (-1), test-en (-1), test-hard (-1)
• Libri & Chinese Dialects — total ?; splits: Libri-Clean (-1), Libri-Other (-1), Sichuan dialect (-1), Jiao-Liao Mandarin (-1), Taiwan Mandarin (-1), Cantonese (-1), Shanghai dialect (-1)

Metrics

• CER (primary) — range: percent
  • Character Error Rate calculated as the edit distance between the ASR transcription of the generated speech and the reference text, divided by the length of the reference text. Lower is better.
• SIM — range: [0, 1]
  • Speaker Similarity computed as the cosine similarity between speaker embeddings extracted from the generated audio and the reference audio using a fine-tuned WavLM-large model. Higher is better.
• EMO — range: [0, 1]
  • Emotion score output by an internal emotion classification model on the generated audio, indicating emotional expressiveness. Higher is better.
• PER — range: percent
  • Phoneme Error Rate calculated via phoneme-level alignment between generated and reference speech. Lower is better.
• NISQA — range: other
  • Objective speech quality score from the NISQA model. Higher is better.
• UTMOS — range: other
  • Objective mean opinion score predicted by the UTMOS model. Higher is better.
• MOS — range: other
  • Subjective Mean Opinion Score from human listeners evaluating audio quality. Higher is better.

Input / output format

Input: Reference text (and optional phoneme sequence for the Phoneme-in ablation) and reference audio (for voice cloning/SIM evaluation).

Output: Synthesized speech audio waveform.

Scoring recipe

def compute_metrics(generated_audio, reference_audio, reference_text, reference_phonemes=None):
    # ASR-based metrics
    transcribed = ASR_model(generated_audio)
    cer = edit_distance(transcribed, reference_text) / len(reference_text)
    wer = edit_distance(transcribed.split(), reference_text.split()) / len(reference_text.split())
    if reference_phonemes:
        per = edit_distance(transcribed_phonemes, reference_phonemes) / len(reference_phonemes)
    # Speaker Similarity
    ref_emb = WavLM_large(reference_audio)
    gen_emb = WavLM_large(generated_audio)
    sim = cosine_similarity(ref_emb, gen_emb)
    # Emotion & Quality
    emo = EmotionClassifier(generated_audio)
    nisqa = NISQA_model(generated_audio)
    utmos = UTMOS_model(generated_audio)
    mos = human_listen_score(generated_audio)
    return {'CER': cer, 'WER': wer, 'PER': per, 'SIM': sim, 'EMO': emo, 'NISQA': nisqa, 'UTMOS': utmos, 'MOS': mos}

Common pitfalls

• SIM is computed using a specific fine-tuned WavLM-large model, not the standard WavLM or other speaker encoders, which limits direct comparability with other benchmarks.
• CER/WER are derived from ASR transcription of generated audio, meaning ASR errors are conflated with TTS pronunciation errors.
• The internal emotion test set and hard-case dataset are proprietary and unreleased, preventing independent verification or reproduction of EMO and PER results.
• The 'test-hard' set contains polyphones and rare words, but its exact composition, size, and filtering criteria are not disclosed.

Evidence (verbatim from paper)

Table[3] reports results on the Seed-TTS-eval benchmark using standard TTS metrics: Character Error Rate (CER) and Word Error Rate (WER) for pronunciation accuracy (lower is better), and Speaker Similarity (SIM, higher is better) measured by calculating the cosine similarity between speaker embeddings extracted using fine-tuned WavLM-large*(chen2022largescaleselfsupervisedspeechrepresentation)*.

Citation

@misc{cui2025gltts,
  title={GLM-TTS Technical Report},
  author={Cui et al. (2025)},
  year={2025},
  note={arXiv:2512.14291}
}

• arXiv: 2512.14291