Singing Voice Conversion Eval

singing-voice-conversion-eval

Learning Singing From Speech — Zhang et al. (2019) (arXiv:1912.10128, 2019)

What this evaluates

Probes a model's ability to convert speech to high-quality singing while preserving the target speaker's timbre, using only normal speech samples. Evaluates both audio naturalness and speaker similarity through subjective listening tests.

Datasets

• Tencent multi-speaker speech corpus (TSP) — total ?; splits: train (-1)
• Tencent singing corpus (TSG) — total ?; splits: train (-1)
• Separate singing corpus (test) — total ?; splits: test (20)

Metrics

• MOS (Naturalness) (primary) — range: other
  • Subjective Mean Opinion Score where 14 human listeners rate audio quality on a 1-to-5 scale (5 = highest).
• MOS (Similarity) — range: other
  • Subjective Mean Opinion Score where 14 human listeners rate timbre similarity between converted singing and target speaker's voice on a 1-to-5 scale (5 = highest).

Input / output format

Input: Source singing audio segments and target speaker's reference audio (speech or singing), conditioned on frame-level f0 and RMSE features.

Output: Converted singing audio waveform.

Scoring recipe

# 1. Generate converted audio for 20 test segments.
# 2. Present to 14 human listeners for subjective listening tests.
# 3. Listeners assign a score from 1 to 5 for Naturalness and Similarity.
# 4. Compute the mean score across all 14 listeners per segment.
# 5. Average the segment means to obtain the final MOS value.

Common pitfalls

• The test corpus is not publicly released, preventing exact reproduction of the 20 test segments.
• Using target speaker's singing samples for conversion yields significantly higher MOS than using only speech samples, which is expected but must be distinguished when evaluating the paper's core claim of speech-to-singing conversion.
• MOS relies on a small panel of 14 subjects, making scores sensitive to listener demographics and listening conditions.

Evidence (verbatim from paper)

Subjective evaluation with Mean Opinion Scores (MOS) is used. A total of 14 subjects have been participated in our listening tests. The scale of MOS is set between 1 to 5 with 5 being the highest score.

Citation

@misc{zhang2019learningsinging,
  title={Learning Singing From Speech},
  author={Zhang et al. (2019)},
  year={2019},
  note={arXiv:1912.10128}
}

• arXiv: 1912.10128