Samoye Svc Eval

samoye-svc-eval

SaMoye: Zero-shot Singing Voice Conversion Model Based on Feature Disentanglement and Enhancement — Zihao Wang et al. (2024) (arXiv:2407.07728, 2024)

What this evaluates

Evaluates zero-shot singing voice conversion by measuring timbre transfer accuracy and audio quality. It probes the model's ability to disentangle content, pitch, and timbre, and generalize to unseen human and non-human (animal) speakers without fine-tuning.

Datasets

• Custom zero-shot test set — total 5; splits: test (5)

Metrics

• MOS-S (primary) — range: [1, 5]
  • Mean Opinion Score on Similarity based on a 5-point Likert scale (1=completely different timbre, 5=same timbre).
• MOS-Q — range: [1, 5]
  • Mean Opinion Score on Quality based on expert evaluation of audio fidelity.
• PESQ — range: [-0.5, 4.5]
  • Perceptual Evaluation of Speech Quality measuring temporal alignment and perceptual filtering.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility measuring comprehensibility of audio.
• NISQA — range: other
  • Non-Intrusive Speech Quality Assessment using a pre-trained deep learning model to predict MOS, noise, coherence, timbre, and loudness.
• SECS — range: [0, 1]
  • Speaker Encoder Cosine Similarity computed between CAM++ embeddings of generated and original audio.

Input / output format

Input: Source singing audio waveform and reference audio waveform (target timbre).

Output: Converted singing audio waveform.

Scoring recipe

import numpy as np
from scipy.spatial.distance import cosine

def compute_secs(pred_wav, gold_wav):
    pred_emb = cam_plus_plus_embed(pred_wav)
    gold_emb = cam_plus_plus_embed(gold_wav)
    return 1.0 - cosine(pred_emb, gold_emb)

def compute_mos_s(ratings):
    return np.mean(ratings)

Common pitfalls

• NISQA is pretrained on speech datasets and may misjudge singing audio quality due to the speech-singing gap.
• SECS relies on CAM++, which is trained on human speech, making it unreliable for non-human/animal timbres.
• Subjective MOS scores can be confounded by overall audio quality; high MOS-Q often correlates with high MOS-S regardless of actual timbre fidelity.

Evidence (verbatim from paper)

The subjective metrics include:

• Mean Opinion Score on Similarity(MOS-S): MOS-S is based on a 5-score Likert scale, where 5 means the same timbre and 1 for a completely different timbre.
• Mean Opinion Score on Quality(MOS-Q): MOS is a widely-used audio or video quality evaluation standard based on expert evaluation. The score of MOS is from 1 to 5, where a higher score means higher quality.

Citation

@misc{wang2024samoye,
  title={SaMoye: Zero-shot Singing Voice Conversion Model Based on Feature Disentanglement and Enhancement},
  author={Zihao Wang et al. (2024)},
  year={2024},
  note={arXiv:2407.07728}
}

• arXiv: 2407.07728