Speech Likelihood Eval

speech-likelihood-eval

Benchmarking Generative Latent Variable Models for Speech — Havtorn et al. (2022) (arXiv:2202.12707, 2022)

What this evaluates

Evaluates the ability of generative latent variable models and autoregressive baselines to model speech audio distributions at varying temporal resolutions. It measures how well models capture intra-frame and inter-frame correlations in audio waveforms by optimizing likelihood objectives.

Datasets

• TIMIT — total ?; splits: train (-1), val (-1), test (-1)
• LibriSpeech — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1)

Metrics

• bits per frame (bpf) (primary) — range: other
  • Negative log-likelihood per audio frame converted to bits. For latent variable models, the one-sample Evidence Lower Bound (ELBO) is reported as an approximation. Lower values indicate better model fit and compression efficiency.

Input / output format

Input: Stacked μ-law encoded PCM audio frames standardized to [-1, 1]. Each input x_t consists of s consecutive waveform frames x̃_{t:t+s}, where stack size s ∈ {1, 64, 256}. Audio is 16-bit depth at 16 kHz sample rate.

Output: Probability distribution (specifically a 10-component Discrete Mixture of Logistics, DMoL) over the discretized audio values for the stacked frames.

Scoring recipe

def compute_bpf(predictions, gold, stack_size, num_frames):
    # 1. Compute log-likelihood log p(x) using the model's output distribution (DMoL)
    # 2. For LVMs, use the one-sample ELBO as the log-likelihood estimate
    log_likelihood = compute_log_prob(predictions, gold)
    # 3. Convert to bits per frame: bpf = -log_p_x / (num_frames * log(2))
    bpf = -log_likelihood / (num_frames * np.log(2))
    # 4. Average bpf across all samples in the split
    return np.mean(bpf)

Common pitfalls

• LVMs report the one-sample ELBO, which is a lower bound on the true log-likelihood; the paper marks these with ≤ in tables, meaning the true likelihood could be better (lower bpf).
• Stack size s drastically changes temporal resolution and likelihood values; models must be compared at identical s to avoid unfair comparisons.
• FLAC compression rates are reported on linear PCM, while models are trained on μ-law encoded audio, making direct compression-to-likelihood comparisons approximate.

Evidence (verbatim from paper)

We report likelihoods in units of bits per frame (bpf) as this yields a more interpretable and comparable likelihood than total likelihood in nats. It also has direct connections with information theory and compression (Shannon, 1948; Townsend et al., 2019). In units of bits per frame, lower is better. For LVMs, we report the one-sample ELBO.

Citation

@misc{havtorn2022benchmarking,
  title={Benchmarking Generative Latent Variable Models for Speech},
  author={Havtorn et al. (2022)},
  year={2022},
  note={arXiv:2202.12707}
}

• arXiv: 2202.12707