Zipvoice Dialog Eval

zipvoice-dialog-eval

ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching — Han Zhu et al. (2025) (arXiv:2507.09318, 2025)

What this evaluates

This benchmark evaluates non-autoregressive spoken dialogue generation models on their ability to produce multi-turn conversational audio that matches input text, maintains speaker identity, and accurately handles turn-taking between two speakers. It probes both objective speech quality metrics and subjective human judgments of coherence and similarity.

Datasets

• test-dialog-zh — total 357; splits: test (357)
• test-dialog-en — total 280; splits: test (280)

Metrics

• cpWER (primary) — range: percent
  • Concatenated minimum permutation word error rate. Computed by concatenating all utterances per speaker for reference and hypothesis, then calculating WER across all possible speaker permutations of the hypothesis. The lowest WER is selected as the final score.
• WER — range: percent
  • Word Error Rate calculated between the transcription of the synthesized dialogue and the input text, disregarding speaker identity.
• cpSIM — range: other
  • Maximum speaker permutation cosine distance between speaker embeddings of generated and prompt speech, computed similarly to cpWER.
• UTMOS — range: other
  • Neural network-based Mean Opinion Score prediction model used to assess overall speech quality.
• RTF — range: other
  • Real-Time Factor measuring inference speed, calculated as the ratio of audio duration to generation time.
• CMOS — range: other
  • Comparative Mean Opinion Score where evaluators judge relative dialogue quality on a scale of [-3, 3].
• SMOS — range: other
  • Similarity Mean Opinion Score where evaluators rate speaker similarity between prompt and generated dialogues on a scale of [0, 5].

Input / output format

Input: Input text condition and reference prompt speech (with speaker embeddings) for a two-speaker dialogue.

Output: Synthesized multi-turn spoken dialogue audio matching the input text and speaker turns.

Scoring recipe

def compute_cpWER(ref_text, hyp_text, n_speakers=2):
    ref_spk = split_by_speaker(ref_text)
    hyp_spk = split_by_speaker(hyp_text)
    ref_concat = ''.join(ref_spk)
    hyp_concat = ''.join(hyp_spk)
    min_wer = float('inf')
    for perm in itertools.permutations(range(n_speakers)):
        permuted_hyp = apply_permutation(hyp_concat, perm)
        wer = calculate_wer(ref_concat, permuted_hyp)
        if wer < min_wer:
            min_wer = wer
    return min_wer

Common pitfalls

• WhisperD has a 30-second segment limit, so cpWER is only evaluated on short English dialogues (test-dialog-en short), not Chinese or longer turns.
• Standard WER ignores speaker attribution; a low WER does not guarantee correct turn-taking, so cpWER must be used to measure speaker identity accuracy.

Evidence (verbatim from paper)

To assess speaker turn-taking accuracy, i.e., whether the correct speaker voice is attributed to each utterance, we adopted the concatenated minimum permutation word error rate (cpWER) [44]. cpWER is computed by first concatenating all utterances per speaker for both reference and hypothesis files, then calculating the WER between the reference and all possible speaker permutations of the hypothesis (e.g., two permutations for two-party dialogues). The lowest WER among these permutations is selected as the final cpWER.

Citation

@misc{zhu2025zipvoicedialog,
  title={ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching},
  author={Han Zhu et al. (2025)},
  year={2025},
  note={arXiv:2507.09318}
}

• arXiv: 2507.09318