Duriansc Svc Eval

duriansc-svc-eval

DurIAN-SC: Duration Informed Attention Network based Singing Voice Conversion System — Liqiang Zhang et al. (2020) (arXiv:2008.03009, 2020)

What this evaluates

Evaluates one-shot singing voice conversion quality by measuring how naturally the converted audio sounds and how closely it matches the target speaker's voice, using only 20 seconds of target speech or singing data.

Datasets

• Database A — total 3600; splits: train (3500), test_in-set (6)
• Database B — total ?; splits: test_out-set (4)

Metrics

• MOS naturalness (primary) — range: [1, 5]
  • Mean Opinion Score on a 1-5 scale (1=worst, 5=best) assessing the naturalness of the converted singing audio.
• MOS similarity — range: [1, 5]
  • Mean Opinion Score on a 1-5 scale (1=worst, 5=best) assessing similarity to the target speaker's voice.

Input / output format

Input: Source singing audio, target speaker d-vector extracted from 20s of speech/singing, frame-level f0 and RMSE conditioning.

Output: Converted singing audio waveform.

Scoring recipe

def evaluate_mos(predictions, gold, testers=10):
    scores_naturalness = []
    scores_similarity = []
    for tester in range(testers):
        for pred, gold in zip(predictions, gold):
            # Human listening test
            n = tester.rate_naturalness(pred) # 1-5
            s = tester.rate_similarity(pred)   # 1-5
            scores_naturalness.append(n)
            scores_similarity.append(s)
    return mean(scores_naturalness), mean(scores_similarity)

Common pitfalls

• MOS scores are averaged over only 10 testers, which may limit statistical reliability and generalizability.
• Out-of-set similarity scores drop significantly because d-vectors are extracted from just 20s of data and the model is not fine-tuned on unseen speakers.
• The LUT baseline cannot be evaluated on out-of-set speakers, preventing a direct comparison for that condition.

Evidence (verbatim from paper)

In the singing voice conversion test, Mean Opinion Scores (MOS) on naturalness and similarity to target speaker are evaluated. The scale of MOS is set between 1 to 5 with 5 representing the best performance and 1 the worst. 10 testers participated in our listening test.

Citation

@misc{zhang2020duriansc,
  title={DurIAN-SC: Duration Informed Attention Network based Singing Voice Conversion System},
  author={Liqiang Zhang et al. (2020)},
  year={2020},
  note={arXiv:2008.03009}
}

• arXiv: 2008.03009