Marco Voice Eval

marco-voice-eval

Marco-Voice Technical Report — Tian et al. (2025) (arXiv:2508.02038, 2025)

What this evaluates

Evaluates a unified neural TTS framework's ability to disentangle speaker identity and emotional style, measuring speaker fidelity, emotional expressiveness, and overall speech quality in both English and Mandarin.

Datasets

• LibriTTS — total ?; splits: test (400)
• AISHELL-3 — total ?; splits: test (400)
• CSEMOTIONS — total ?; splits: test (700)

Metrics

• Speaker similarity — range: [-1, 1]
  • Cosine similarity between speaker embeddings extracted by a pre-trained speaker model from the reference and generated audio.
• Emotional expressiveness (primary) — range: [1, 5]
  • Mean rating on a 5-point Likert scale provided by human listeners assessing the emotional realism of the synthesized speech.
• MOS — range: [1, 5]
  • Mean Opinion Score from human listeners evaluating overall speech quality.
• Whisper-WER — range: [0, 100]
  • Word Error Rate computed by the Whisper ASR model comparing the transcribed generated audio against the ground-truth text prompt.
• DNS-MOS — range: [1, 5]
  • Deep Noise Suppression MOS, an objective neural metric estimating speech quality and naturalness.

Input / output format

Input: Text prompts (Chinese or English) paired with reference audio clips for speaker identity and emotional style conditioning.

Output: Synthesized audio waveform.

Scoring recipe

def compute_metrics(reference_audio, generated_audio, text_prompt):
    # Speaker Similarity
    ref_emb = speaker_model.encode(reference_audio)
    gen_emb = speaker_model.encode(generated_audio)
    speaker_sim = cosine_similarity(ref_emb, gen_emb)

    # Human Evaluations (Emotional expressiveness & MOS)
    human_ratings = collect_human_ratings(generated_audio, scale=5)
    emotion_score = mean(human_ratings['emotion'])
    mos_score = mean(human_ratings['quality'])

    # Automatic Metrics
    wer = whisper_asr.transcribe(generated_audio)
    wer = word_error_rate(wer, text_prompt)
    dns_mos = dns_mos_model.predict(generated_audio)

    return {
        'speaker_similarity': speaker_sim,
        'emotional_expressiveness': emotion_score,
        'mos': mos_score,
        'whisper_wer': wer,
        'dns_mos': dns_mos
    }

Common pitfalls

• Relying solely on automatic metrics (e.g., DNS-MOS or Whisper-WER) to judge emotional expressiveness, as the paper explicitly states human evaluation is the primary measure for emotion.
• Speaker similarity scores are highly dependent on the specific pre-trained speaker embedding model used; results are not directly comparable across different embedding backbones.
• Whisper-WER may penalize emotionally distorted phonetics that are intentional in emotional speech synthesis, conflating emotional prosody with transcription errors.

Evidence (verbatim from paper)

We evaluated our system mainly based on human evaluation with additional automatic metrics for analysis to address the challenges for evaluating emotional speech generation with voice cloning: Speaker similarity was measured using a pre-trained speaker model that computes cosine similarity between speaker embeddings. Emotional expressiveness was evaluated through human ratings on a 5-point Likert scale. Overall speech quality was assessed using mean opinion scores (MOS) from human listeners, as well as objective metrics including Whisper-WER and DNS-MOS.

Citation

@misc{tian2025marcovoice,
  title={Marco-Voice Technical Report},
  author={Tian et al. (2025)},
  year={2025},
  note={arXiv:2508.02038}
}

• arXiv: 2508.02038