Vocbench Eval

vocbench-eval

VocBench: A Neural Vocoder Benchmark for Speech Synthesis — AlBadawy et al. (2021) (arXiv:2112.03099, 2021)

What this evaluates

Evaluates the audio synthesis quality, computational efficiency, and speaker generalization of neural vocoders across autoregressive, GAN-based, and diffusion-based architectures. It probes how well models preserve waveform fidelity and spectrogram structure while balancing inference speed and training complexity.

Datasets

• LJ Speech — total 13100; splits: train (13070), val (10), test (20)
• LibriTTS — total ?; splits: train (-1), val (-1), test (-1)
• VCTK — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MOS (primary) — range: [0, 5]
  • 5-scale subjective numerical measure of human-judged overall quality after listening to a synthesized sample.
• SSIM — range: [0, 1]
  • Quantitative metric measuring similarity between synthetic and real spectrograms in the frequency domain.
• FAD — range: other
  • Distance between two multivariate Gaussians estimated on feature embeddings (background vs evaluation) using a VGG audio classifier.
• LS-MSE — range: other
  • Mean squared error between ground truth and generated log-mel spectrograms.
• PSNR — range: other
  • Ratio of peak signal power to noise power in dB, computed in the frequency domain where peak=1 and noise=LS-MSE.

Input / output format

Input: 80-dimensional log-mel spectrogram features extracted from 24 kHz speech signals (40 ms Hanning window, 12.5 ms frame shift, 1024-point FFT, 0-12 kHz cutoffs, log dynamic range compression, min-max normalized).

Output: Synthesized audio waveform (24 kHz sampling rate).

Scoring recipe

def compute_vocbench_metrics(gt_audios, pred_audios):
    metrics = {'MOS': [], 'SSIM': [], 'LS-MSE': [], 'PSNR': [], 'FAD': []}
    for gt, pred in zip(gt_audios, pred_audios):
        gt_spec = extract_log_mel(gt)
        pred_spec = extract_log_mel(pred)
        metrics['LS-MSE'].append(mse(gt_spec, pred_spec))
        metrics['SSIM'].append(ssim(gt_spec, pred_spec))
        metrics['PSNR'].append(10 * log10(1.0 / metrics['LS-MSE'][-1]))
        metrics['FAD'].append(vgg_embeddings(pred))
    metrics['MOS'] = average_human_ratings(gt_audios, pred_audios)
    metrics['FAD'] = frechet_distance(metrics['FAD'], bg_embeddings)
    return {k: mean(v) for k, v in metrics.items()}

Common pitfalls

• Different vocoders require different input preprocessing (e.g., μ-law compression for WaveNet on LJ/VCTK vs raw waveform for LibriTTS), so failing to standardize inputs breaks fair comparison.
• RTF (Real Time Factor) is excluded for autoregressive models due to extreme slowness, making speed benchmarks incomplete for that architecture class.
• FAD and MOS do not always correlate; a model can achieve the best FAD score but second-best MOS, requiring both to be reported for a complete evaluation.

Evidence (verbatim from paper)

We adopt the following metrics is this study: Mean Opinion Score (MOS) is a subjective numerical measure of the human-judged overall quality after listening to a sample. We use 5-scale MOS to assess the quality for the synthesized speech sample. ... Fréchet Audio Distance (FAD) measures the quality and diversity of the generated samples. ... Log-mel Spectrogram Mean Squared Error (LS-MSE) is computed between the ground truth spectrogram sample and a generated one. ... Peak Signal-to-Noise Ratio (PSNR) is the ratio of the power of a peak signal, which is the magnitude of the best-case output of a signal to the power of the noise at the peak measured in dB. We apply PSNR computation in the frequency domain, where the peak signal of the output is 1 and the distorting noise is represented by LS-MSE.

Citation

@misc{albadawy2021vocbench,
  title={VocBench: A Neural Vocoder Benchmark for Speech Synthesis},
  author={AlBadawy et al. (2021)},
  year={2021},
  note={arXiv:2112.03099}
}

• arXiv: 2112.03099