Vcc2018 Spoke Eval

vcc2018-spoke-eval

Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech — Luong et al. (2019) (arXiv:1909.06532, 2019)

What this evaluates

Evaluates non-parallel voice conversion by measuring how effectively a system transfers a source speaker's identity to a target speaker while preserving linguistic content. It probes intra-lingual conversion fidelity and cross-lingual adaptation using subjective human ratings.

Datasets

• VCC2018 (Voice Conversion Challenge 2018) — total ?; splits: test (140)
• VCTK — total ?; splits: train (-1)

Metrics

• Quality (primary) — range: [1, 5]
  • Mean subjective rating score assigned by human listeners to evaluate the overall audio quality of the converted speech.
• Similarity — range: [1, 5]
  • Mean subjective rating score assigned by human listeners to evaluate how closely the converted speech matches the target speaker's voice.

Input / output format

Input: Source speaker speech utterances (English or Japanese) provided as audio input.

Output: Converted speech waveform matching the target speaker's voice characteristics.

Scoring recipe

def compute_metrics(predictions, gold):
    quality_ratings = []
    similarity_ratings = []
    for pred_audio, target_id in zip(predictions, gold):
        # Human listeners rate on a standard scale (typically 1-5)
        quality_ratings.append(listener.rate_quality(pred_audio))
        similarity_ratings.append(listener.rate_similarity(pred_audio, target_id))
    return {
        'Quality': sum(quality_ratings) / len(quality_ratings),
        'Similarity': sum(similarity_ratings) / len(similarity_ratings)
    }

Common pitfalls

• Relies exclusively on subjective human listening tests rather than objective acoustic metrics like PESQ or STOI.
• Cross-lingual evaluation uses custom in-house bilingual speakers instead of standard benchmark datasets.
• Explicitly ignores the provided source speaker training data from VCC2018, which may limit direct comparison with baselines that use it.

Evidence (verbatim from paper)

To validate the proposed method, we reenact the SPOKE task of the Voice Conversion Challenge 2018 (VCC2018). We build the VC system for 4 target speakers (2 males and 2 females) using 81 utterances per person. We then evaluate the SPOKE task using the speech of 4 source speakers (2 males and 2 females); each speaker contribute 35 utterances. Table 1: Detailed subjective results for SPOKE task. (a) Quality (b) Similarity

Citation

@misc{luong2019bootstrapping,
  title={Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech},
  author={Luong et al. (2019)},
  year={2019},
  note={arXiv:1909.06532}
}

• arXiv: 1909.06532