Breezyvoice Tts Eval

breezyvoice-tts-eval

BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights — Hsu et al. (2025) (arXiv:2501.17790, 2025)

What this evaluates

Evaluates the phonetic accuracy, audio quality, and speaker similarity of a Taiwanese Mandarin TTS system, with a focus on voice cloning robustness and code-switching scenarios. The benchmark probes the model's ability to handle long-tail speaker variability and context-dependent pronunciation ambiguities in both monolingual and bilingual contexts.

Datasets

• FormosaSpeech (subset) — total 15; splits: test (15)
• Spontaneous Recordings — total 100; splits: test (100)
• Traditional Chinese Monologue Dataset (TCMD) — total ?; splits: test (-1)
• Traditional Chinese Code-switching Dataset (TCCSD) — total ?; splits: test (-1)

Metrics

• PER (primary) — range: percent
  • Phone Error Rate; measures the normalized edit distance between predicted and reference phoneme sequences. Formula not explicitly provided in the text.
• Speaker Similarity — range: [0, 1]
  • Quantifies the acoustic similarity between generated and reference speech, typically via embedding cosine similarity. Specific implementation details are not provided in the text.

Input / output format

Input: Traditional Chinese text (monologue or code-switching), conditioning speech utterances (5–15 seconds from diverse speakers), and optionally augmented Mandarin phonetic symbols.

Output: Generated audio waveform (derived from mel-spectrograms via a static algorithmic vocoder).

Scoring recipe

def evaluate(predictions, gold):
    per_scores = []
    sim_scores = []
    for pred_audio, ref_audio, ref_text in zip(predictions, gold['audio'], gold['text']):
        # Compute PER between generated and reference phonemes
        pred_phones = extract_phonemes(pred_audio)
        ref_phones = extract_phonemes(ref_audio)
        per_scores.append(edit_distance(pred_phones, ref_phones) / len(ref_phones))
        # Compute Speaker Similarity between generated and reference audio
        sim_scores.append(cosine_similarity(embed(pred_audio), embed(ref_audio)))
    return {'PER': mean(per_scores), 'Speaker Similarity': mean(sim_scores)}

Common pitfalls

• Comparing against proprietary TTS systems that rely on preset voices without accounting for their lack of cloning capabilities, leading to unfair audio quality comparisons.
• Ignoring code-switching scenarios, which are critical for evaluating Taiwanese Mandarin TTS robustness in real-world contexts involving entity names and technical jargon.
• Failing to use a diverse range of speakers for voice cloning evaluations, which can overestimate speaker similarity and mask long-tail variability issues.

Evidence (verbatim from paper)

For voice cloning evaluations, we evaluate on PER and speaker similarity using samples generated from a diverse range of speakers to gauge the robustness of our system. ... Phonetic accuracy and prosodic appropriateness are the primary evaluation criteria.

Citation

@misc{hsu2025breezyvoice,
  title={BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights},
  author={Hsu et al. (2025)},
  year={2025},
  note={arXiv:2501.17790}
}

• arXiv: 2501.17790