Covocheval

covocheval

The ISCSLP 2024 Conversational Voice Clone (CoVoC) Challenge: Tasks, Results and Findings — Xia et al. (2024) (arXiv:2411.00064, 2024)

What this evaluates

Evaluates zero-shot conversational voice cloning systems on their ability to generate natural, expressive speech that matches a target speaker's timbre and spontaneous style without prior training on the target speaker. It measures pronunciation accuracy, speaker similarity, and subjective qualities like naturalness, quality, and spontaneous style.

Datasets

• HQ-Conversations / CoVoC Test Prompts — total ?; splits: test (-1)

Metrics

• CER — range: percent
  • Character Error Rate computed between the ground truth transcript and the transcript recognized by an external ASR model (paraformer-large) from the synthesized audio.
• SIM — range: [-1, 1]
  • Cosine similarity between speaker embeddings extracted from the reference speech and the generated speech using the Resemblyzer tool.
• SN — range: [1, 5]
  • Mean Opinion Score (1-5) for speech naturalness, assessing pronunciation correctness, ambiguity, tone changes, and pause naturalness.
• SQ — range: [1, 5]
  • Mean Opinion Score (1-5) for speech quality, assessing electronic distortion and voice clarity.
• SS — range: [1, 5]
  • Mean Opinion Score (1-5) for speaker similarity, assessing timbre and speaking style match to the target speaker.
• SSS — range: [1, 5]
  • Mean Opinion Score (1-5) for speech spontaneous style, assessing colloquial words, laughter, non-rhythmic pauses, stress, and rhythm.
• FS (primary) — range: [1, 5]
  • Final Score computed as the unweighted average of the four MOS scores: FS = 0.25×SN + 0.25×SQ + 0.25×SS + 0.25×SSS.

Input / output format

Input: Reference audio clip (target speaker) and target text transcript.

Output: Synthesized audio waveform matching the target text in the target speaker's voice.

Scoring recipe

def compute_metrics(reference_audio, predicted_audio, gold_transcript):
    # Objective
    cer = compute_cer(gold_transcript, asr_model(predicted_audio))
    sim = cosine_similarity(resemblyzer_embed(reference_audio), resemblyzer_embed(predicted_audio))
    # Subjective (averaged across 10 raters)
    sn, sq, ss, sss = [rater_score(sample) for rater_score in raters]
    fs = 0.25 * sn + 0.25 * sq + 0.25 * ss + 0.25 * sss
    return {"CER": cer, "SIM": sim, "SN": sn, "SQ": sq, "SS": ss, "SSS": sss, "FS": fs}

Common pitfalls

• CER requires an external ASR model to transcribe the synthesized audio before computing error; it is not a direct string-to-string comparison.
• All audio samples must be resampled to 16 kHz before objective metric computation.
• The Final Score (FS) is a simple unweighted average of the four MOS aspects, despite the formula notation implying a weighted sum.

Evidence (verbatim from paper)

The objective evaluation consisted of two aspects: pronunciation accuracy and timbre similarity. We employ Character Error Rate (CER) and cosine similarity for evaluation: Character Error Rate: CER is computed between the ground truth transcript and the recognized transcript. We use an open-source paraformer-large model to recognize the synthesized speech into the corresponding transcription. Speaker Similarity (SIM): We employ the Resemblyzer tool to extract speaker embedding and compute the cosine similarity between the reference speech and generated speech.

Citation

@misc{xia2024covocheval,
  title={The ISCSLP 2024 Conversational Voice Clone (CoVoC) Challenge: Tasks, Results and Findings},
  author={Xia et al. (2024)},
  year={2024},
  note={arXiv:2411.00064}
}

• arXiv: 2411.00064