Singverse Eval

singverse-eval

SingVERSE: A Diverse, Real-World Benchmark for Singing Voice Enhancement — Jiang et al. (2025) (arXiv:2509.20969, 2025)

What this evaluates

Evaluates singing voice enhancement models on real-world acoustic scenarios, measuring their ability to improve perceptual quality and content intelligibility of degraded singing vocals without degrading speech capabilities.

Datasets

• SingVERSE — total 3971; splits: test (3971)

Metrics

• perceptual quality (primary) — range: not specified
  • Not explicitly defined in the provided section; measures the fidelity and naturalness of the enhanced singing voice relative to the clean reference.
• content intelligibility — range: not specified
  • Not explicitly defined in the provided section; measures the clarity and understandability of the enhanced singing lyrics relative to the clean reference.

Input / output format

Input: Degraded real-world singing audio clips from 19 diverse acoustic environments, paired with studio-quality clean references.

Output: Enhanced singing audio clips.

Scoring recipe

def evaluate(predictions, gold):
    # predictions: enhanced audio, gold: clean reference
    perceptual_score = compute_perceptual_quality(predictions, gold)
    intelligibility_score = compute_intelligibility(predictions, gold)
    return perceptual_score, intelligibility_score

Common pitfalls

• Models trained on synthetic data often fail to generalize to the diverse real-world acoustic scenarios covered in the benchmark.
• Optimizing solely for perceptual quality can degrade content intelligibility, highlighting a critical trade-off that must be balanced during evaluation.

Evidence (verbatim from paper)

A controlled experiment reveals a statistically significant domain mismatch when speech enhancement models are applied to singing, with performance drops in perceptual quality and intelligibility across all models.

Citation

@misc{jiang2025singverse,
  title={SingVERSE: A Diverse, Real-World Benchmark for Singing Voice Enhancement},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2509.20969}
}

• arXiv: 2509.20969