Llama Vits Tts Eval

llama-vits-tts-eval

Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness — Feng et al. (2024) (arXiv:2404.06714, 2024)

What this evaluates

Evaluates the naturalness, intelligibility, and emotional expressiveness of a non-autoregressive TTS model enhanced with LLM-derived semantic embeddings. Probes how well semantic tokens from Llama2 versus BERT improve acoustic quality and emotion similarity compared to baselines.

Datasets

• LJSpeech — total ?; splits: train (-1), test (-1)
• 1-hour LJSpeech — total ?; splits: train (-1), test (-1)
• EmoV_DB_bea_sem — total ?; splits: train (-1), test (51)

Metrics

• ESMOS (primary) — range: [1, 5]
  • Emotion Similarity Mean Opinion Score. Participants rate emotion similarity to ground truth on a 5-point scale: 1 (Bad Match) to 5 (Excellent Match).
• UTMOS — range: [1, 5]
  • UTokyo-SaruLab Mean Opinion Score. A neural MOS prediction network trained on Blizzard and Voice Conversion Challenge data.
• MCD — range: dB
  • Mel-Cepstral Distortion. Measures spectral distortion between synthesized and reference speech in decibels.
• CER — range: percent
  • Character Error Rate. Computed using Whisper-large to transcribe synthesized audio and compare against ground truth text.
• WER — range: percent
  • Word Error Rate. Computed using Whisper-large to transcribe synthesized audio and compare against ground truth text.

Input / output format

Input: Text transcript or phoneme sequence, optionally augmented with global or sequential semantic embeddings extracted from Llama2 or BERT models.

Output: Synthesized speech waveform.

Scoring recipe

# ESMOS (Subjective)
esmos_scores = []
for sample in test_samples:
    score = mturk_participant_rating(sample.synthesized_audio, sample.gt_audio, scale=1-5)
    esmos_scores.append(score)
esmos = mean(esmos_scores)

# UTMOS (Objective)
utmos_scores = [speechmos_model.predict(audio) for audio in test_audios]
utmos = mean(utmos_scores)

# MCD (Objective)
mcd_values = [mel_cepstral_distortion(ref, synth) for ref, synth in zip(ref_audios, synth_audios)]
mcd = mean(mcd_values)

# CER/WER (Objective via Whisper-large)
transcriptions = whisper_large.transcribe(test_audios)
cer = character_error_rate(transcriptions, ground_truth_texts)
wer = word_error_rate(transcriptions, ground_truth_texts)

Common pitfalls

• ESMOS instructions explicitly tell participants to ignore voice quality, style, and audio quality, focusing solely on emotiveness; failing to enforce this skews results.
• Training protocols differ by dataset size: LJSpeech variants train from scratch for 100k steps, while EmoV_DB_bea_sem fine-tunes a 100k-step LJSpeech checkpoint for 150k steps, making step-count comparisons invalid across datasets.
• Objective metrics rely on specific external toolchains (SpeechMOS for UTMOS, ESPnet for MCD, Whisper-large for CER/WER) rather than native model outputs, requiring strict version control for reproducibility.

Evidence (verbatim from paper)

In subjective evaluation, we conduct Emotion Similarity Mean Opinion Score (ESMOS) experiments to evaluate emotion similarity for EmoV_DB_bea_sem. ... In objective evaluation, we utilize UTokyo-SaruLab Mean Opinion Score (UTMOS), Mel-Cepstral Distortion (MCD), and speech recognition performance measured by Character Error Rate (CER) and Word Error Rate (WER).

Citation

@misc{feng2024llamavits,
  title={Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness},
  author={Feng et al. (2024)},
  year={2024},
  note={arXiv:2404.06714}
}

• arXiv: 2404.06714