Seed Tts Eval

seed-tts-eval

X-VC: Zero-shot Streaming Voice Conversion in Codec Space — Zheng et al. (2026) (arXiv:2604.12456, 2026)

What this evaluates

Evaluates zero-shot voice conversion systems on linguistic preservation, speaker identity retention, and audio naturalness across English, Chinese, and cross-lingual settings. It also measures computational efficiency and latency for both streaming and offline inference modes.

Datasets

• Seed-TTS-Eval — total ?; splits: test-en (1000), test-zh (1000), cross-en-zh (1000), cross-zh-en (1000)

Metrics

• WER (%) (primary) — range: percent
  • Word Error Rate measuring linguistic preservation. Computed using Whisper-large-v3 for English and Paraformer-zh for Chinese.
• SIM — range: [-1, 1]
  • Speaker Similarity computed as the cosine similarity between speaker embeddings extracted by a WavLM-based model for the converted speech and the target reference.
• UTMOS — range: [1, 5]
  • Objective estimate of speech naturalness without human annotations, predicted by the UTMOS model.
• RTF — range: [0, inf)
  • Real-Time Factor calculated as the ratio of total end-to-end processing time to the input audio duration.
• SMOS — range: [1, 5]
  • Speaker Similarity Mean Opinion Score from a subjective test with 15 listeners rating on a 5-point scale (1: completely different, 5: identical speaker).

Input / output format

Input: Source speech audio and reference speech audio (target speaker).

Output: Converted speech audio.

Scoring recipe

def evaluate(predictions, gold_refs, langs):
    wer_scores = []
    for pred, lang in zip(predictions, langs):
        asr = whisper_large_v3 if lang == 'en' else paraformer_zh
        wer_scores.append(asr.word_error_rate(pred))
    sim_scores = [cosine_similarity(wavlm_embed(pred), wavlm_embed(ref)) 
                  for pred, ref in zip(predictions, gold_refs)]
    utmos_scores = [utmos_predict(pred) for pred in predictions]
    rtf_scores = [processing_time(pred) / audio_duration(pred) for pred in predictions]
    return {
        'WER (%)': mean(wer_scores),
        'SIM': mean(sim_scores),
        'UTMOS': mean(utmos_scores),
        'RTF': mean(rtf_scores)
    }

Common pitfalls

• MeanVC's offline RTF is measured using a chunkwise inference script rather than full-utterance processing, which inflates latency compared to true offline RTF.
• WER is computed using different ASR models for English (Whisper-large-v3) and Chinese (Paraformer-zh), making cross-lingual WER comparisons potentially biased.
• SMOS relies on a small panel of 15 listeners and random sample ordering, which may limit statistical reliability and introduce listener bias.

Evidence (verbatim from paper)

To assess speech quality, we use Word Error Rate (WER) to measure linguistic preservation, employing Whisper-large-v3 for English and Paraformer-zh for Chinese. Speaker identity consistency is evaluated via Speaker Similarity (SIM), computed as the cosine similarity between speaker embeddings extracted by a WavLM-based model, while UTMOS is utilized to objectively estimate speech naturalness without human annotations. For inference efficiency, all metrics are measured end-to-end on a single NVIDIA RTX 3090 GPU.

Citation

@misc{zheng2026xvc,
  title={X-VC: Zero-shot Streaming Voice Conversion in Codec Space},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2604.12456}
}

• arXiv: 2604.12456