qhjqhj00 voicebank-demand-sep-eval
Separate Anything You Describe — Liu et al. (2023) (arXiv:2308.05037, 2023)
What this evaluates
Evaluates zero-shot language-queried speech enhancement by isolating clean speech from noisy backgrounds. The benchmark tests the model's ability to enhance speech using a fixed text query.
Datasets
- Voicebank-DEMAND — total 824; splits: test (824)
Metrics
PESQ(primary) — range: score- Perceptual Evaluation of Speech Quality, a standardized objective measure of speech quality.
CSIG— range: score- MOS predictor of signal distortion, part of the P.835 standard.
CBAK— range: score- MOS predictor of background-noise intrusiveness, part of the P.835 standard.
COVL— range: score- MOS predictor of overall signal quality, part of the P.835 standard.
SSNR— range: dB- Segmental signal-to-noise ratio, measures SNR over short time frames.
Input / output format
Input: A noisy speech utterance (mixed at 15, 10, 5, or 0 dB SNR) paired with the fixed text query 'Speech'.
Output: Enhanced/cleaned speech waveform corresponding to the target utterance.
Scoring recipe
def score(pred, gold):
pesq = compute_pesq(pred, gold)
csig = compute_csig(pred, gold)
cbak = compute_cbak(pred, gold)
covl = compute_covl(pred, gold)
ssnr = compute_ssnr(pred, gold)
return pesq, csig, cbak, covl, ssnr
Common pitfalls
- All audio clips are resampled to 16 kHz for fair comparison with prior speech enhancement systems.
- The text query is fixed to 'Speech' rather than being a natural language description of the speaker or content.
Evidence (verbatim from paper)
The test set of the Voicebank-DEMAND dataset includes a total of 824 utterances, which is used to evaluate the zero-shot performance of our model on speech enhancement. ... We use 'Speech' as the input text query to perform speech enhancement. ... we apply the perceptual evaluation of speech quality (PESQ), mean opinion score (MOS) predictor of signal distortion (CSIG), MOS predictor of background-noise intrusiveness (CBAK), MOS predictor of overall signal quality (COVL) and segmental signal-to-ratio noise (SSNR) for evaluation.
Citation
@misc{liu2023separate,
title={Separate Anything You Describe},
author={Liu et al. (2023)},
year={2023},
note={arXiv:2308.05037}
}
- arXiv: 2308.05037