Svcd Eval

svcd-eval

Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding — Wang et al. (2021) (arXiv:2110.04754, 2021)

What this evaluates

This protocol evaluates a singing voice conversion model's ability to transform a source singer's voice into a target singer's timbre while preserving musical naturalness and pitch accuracy. It measures both subjective perceptual quality and objective acoustic fidelity using human ratings and signal processing metrics.

Datasets

• VCTK — total ?; splits: train (-1), val (-1), test (-1)
• NUS-48E — total ?; splits: train (-1), val (-1), test (20)

Metrics

• MOS (Naturalness) (primary) — range: [1, 5]
  • 1-5 Mean Opinion Score rating the naturalness of the converted singing audio, where 1 is bad and 5 is excellent.
• MOS (Similarity) (primary) — range: [1, 5]
  • 1-5 Mean Opinion Score rating the timbre-style similarity of the converted audio to the target singer.
• NCC — range: [0, 1]
  • Normalized cross-correlation between the pitch contours of the source and converted singing audios to measure pitch matching accuracy.
• COS-SIM — range: [0, 1]
  • Cosine similarity between d-vectors of the converted and target audio, extracted using a pretrained speaker verification model (Resemblyzer).

Input / output format

Input: Source singing audio waveform and target singer reference/timbre specification.

Output: Converted singing audio waveform.

Scoring recipe

def compute_metrics(predictions, gold):
    mos_naturalness = human_rate(predictions, 'naturalness', scale=1-5)
    mos_similarity = human_rate(predictions, 'timbre_similarity', scale=1-5)
    pitch_src = extract_pitch(gold)
    pitch_pred = extract_pitch(predictions)
    ncc = normalized_cross_correlation(pitch_src, pitch_pred)
    dvec_tgt = extract_dvector(gold)
    dvec_pred = extract_dvector(predictions)
    cos_sim = cosine_similarity(dvec_tgt, dvec_pred)
    return {'mos_naturalness': mos_naturalness, 'mos_similarity': mos_similarity, 'ncc': ncc, 'cos_sim': cos_sim}

Common pitfalls

• MOS scores are highly dependent on the specific panel of 15 music professionals; results are not directly comparable across different papers without identical raters.
• NCC only measures pitch contour alignment, not overall audio quality or timbre fidelity.
• The test set is small (20 segments from 10 singers), making statistical significance of MOS scores sensitive to segment selection.

Evidence (verbatim from paper)

For subjective evaluation, we select 1-5 mean opinion score (MOS) (1-bad, 2-poor, 3-fair, 4-good, 5-excellent) for both singing naturalness and timbre-style similarity with the target singer. For objective evaluation, we select normalized cross-correlation (NCC) to measure the pitch matching degree between the source and converted singing audios.

Citation

@misc{wang2021singing,
  title={Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2110.04754}
}

• arXiv: 2110.04754