Svcc23 Eval

svcc23-eval

The Singing Voice Conversion Challenge 2023 — Huang et al. (2023) (arXiv:2306.14422, 2023)

What this evaluates

Evaluates singing voice conversion systems on in-domain (singing-to-singing) and cross-domain (speech-to-singing) speaker conversion. It probes the model's ability to preserve target speaker identity and musical prosody while converting source audio to the target voice.

Datasets

• SVCC 2023 — total ?; splits: train (-1), test (-1); repo https://github.com/lesterphillip/SVCC23_FastSVC

Metrics

• perceptual quality (subjective evaluation) (primary) — range: other
  • Human listeners rate converted audio on naturalness and target speaker similarity using subjective listening tests.
• objective metrics — range: [0, 1]
  • Automatic computational measures used to assess audio quality and speaker similarity, though noted to correlate weakly with human perception.

Input / output format

Input: Source audio (singing for Task 1, speech for Task 2) and target speaker reference audio (singing for Task 1, speech for Task 2).

Output: Converted singing audio matching the target speaker's voice characteristics and the source audio's musical content.

Scoring recipe

# Pseudo-code for SVCC 2023 evaluation
predictions = model.convert(source_audio, target_ref_audio)
# Subjective evaluation (primary)
naturalness_rating = human_listen_test(predictions, criteria="naturalness")
similarity_rating = human_listen_test(predictions, criteria="speaker_similarity")
# Objective evaluation
obj_score = compute_objective_metrics(predictions, gold_audio)
# Aggregate results across in-domain and cross-domain tasks
final_score = average(naturalness_rating, similarity_rating, obj_score)

Common pitfalls

• Cross-domain SVC is significantly harder than in-domain because models lack target singing style data.
• Objective metrics do not strongly correlate with human perceptual quality, so relying solely on them is misleading.

Evidence (verbatim from paper)

Despite top systems achieving human-level naturalness, no model matched target speaker similarity in either task, with cross-domain SVC being significantly harder. Objective metrics failed to correlate strongly with perceptual quality, highlighting the limitations of current evaluation frameworks in capturing subjective fidelity in singing voice conversion.

Citation

@misc{huang2023svcc23,
  title={The Singing Voice Conversion Challenge 2023},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2306.14422}
}

• arXiv: 2306.14422