Svs Eval

svs-eval

Singing Voice Data Scaling-up: An Introduction to ACE-Opencpop and ACE-KiSing — Shi et al. (2024) (arXiv:2401.17619, 2024)

What this evaluates

Evaluates the acoustic and perceptual quality of singing voice synthesis (SVS) models trained on large-scale multi-singer corpora. It probes direct synthesis capability, cross-domain transfer learning, and data augmentation via joint training.

Datasets

• ACE-Opencpop — total 130; splits: test (-1)
• ACE-KiSing — total 32; splits: test (-1)

Metrics

• MCD — range: other
  • Mel cepstral distortion measuring the spectral envelope difference between synthesized and ground truth audio.
• S. Acc. — range: percent
  • Semitone accuracy measuring the percentage of correctly synthesized pitches within a semitone tolerance.
• F0 RMSE — range: other
  • Logarithmic F0 root mean square error measuring the fundamental frequency deviation between predicted and reference audio.
• SECS — range: [0, 1]
  • Speaker embedding cosine similarity computed using a pre-trained Rawnet3-based extractor to measure speaker identity preservation.
• MOS (primary) — range: [1, 5]
  • Mean Opinion Score on a 5-point scale (1=unreasonable singing, 5=natural singing comparable to human performance), averaged over 30 listeners rating 30 randomly selected samples.

Input / output format

Input: Musical score (pitch, duration, lyrics) and singer ID embedding.

Output: Synthesized audio waveform (resampled to 24 kHz for evaluation).

Scoring recipe

# Resample all predictions to 24kHz for fair objective comparison
pred_24k = resample(predictions, target_sr=24000)
gold_24k = resample(gold, target_sr=24000)

mcd = compute_mcd(pred_24k, gold_24k)
s_acc = compute_semitone_accuracy(pred_24k, gold_24k)
f0_rmse = compute_log_f0_rmse(pred_24k, gold_24k)
secs = compute_speaker_embedding_cosine_similarity(pred_24k, gold_24k)

# Subjective MOS evaluation
samples = random.sample(predictions, 30)
ratings = []
for sample in samples:
    ratings.extend([listener_score(sample) for _ in range(30)])
mos = mean(ratings)

Common pitfalls

• Failing to resample all generated audio to 24 kHz before computing objective metrics (MCD, S. Acc., F0 RMSE), which causes unfair comparisons due to differing model output sampling rates.
• Ignoring domain shift effects in transfer learning; ACE-Opencpop is Mandarin Pop while Kiritan is Japanese Animation, leading to metric discrepancies where MCD may not improve while MOS does.
• Over-relying on F0 RMSE for melisma-heavy datasets; subjective MOS and S. Acc. better capture perceptual quality when complex vocal techniques are present.

Evidence (verbatim from paper)

In our study, we conduct both objective and subjective evaluations on the synthesized singing voice samples. For the objective assessment, we utilize Mel cepstral distortion (MCD), semitone accuracy (S. Acc.), and logarithmic F0 root mean square error (F0 RMSE) as our metrics, consistent with the standards in previous research [11, 16, 35, 36]. Given that the two SVS models employed have different output sampling rates, we resample all generated singing voices to 24kHz for the above metrics calculation to ensure a fair comparison. ... In the subjective evaluation, we conduct a Mean Opinion Score (MOS) test to evaluate the perceptual quality of the synthesized voices. We randomly select 30 synthesized samples from each system for this test. These samples are then evaluated by 30 listeners using a 5-point scale, where 1 indicates "unreasonable singing" and 5 signifies "natural singing comparable to human performance".

Citation

@misc{shi2024aceopencpop,
  title={Singing Voice Data Scaling-up: An Introduction to ACE-Opencpop and ACE-KiSing},
  author={Shi et al. (2024)},
  year={2024},
  note={arXiv:2401.17619}
}

• arXiv: 2401.17619