# Speech Likelihood Eval

> Evaluates the ability of generative latent variable models and autoregressive baselines to model speech audio distributions at varying temporal resolutions. It measures how well models capture intra-frame and inter-frame correlations in audio waveforms by optimizing likelihood objectives. Use when the user wants to benchmark on TIMIT, LibriSpeech, or asks about evaluating this task. Reports bits per frame (bpf).

- Skill: `qhjqhj00/speech-likelihood-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/speech-likelihood-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/speech-likelihood-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/speech-likelihood-eval

---


# speech-likelihood-eval

> Benchmarking Generative Latent Variable Models for Speech — Havtorn et al. (2022) (arXiv:2202.12707, 2022)

## What this evaluates

Evaluates the ability of generative latent variable models and autoregressive baselines to model speech audio distributions at varying temporal resolutions. It measures how well models capture intra-frame and inter-frame correlations in audio waveforms by optimizing likelihood objectives.

## Datasets

- **TIMIT** — total ?; splits: train (-1), val (-1), test (-1)
- **LibriSpeech** — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1)

## Metrics

- `bits per frame (bpf)` **(primary)** — range: other
  - Negative log-likelihood per audio frame converted to bits. For latent variable models, the one-sample Evidence Lower Bound (ELBO) is reported as an approximation. Lower values indicate better model fit and compression efficiency.

## Input / output format

**Input**: Stacked μ-law encoded PCM audio frames standardized to [-1, 1]. Each input x_t consists of s consecutive waveform frames x̃_{t:t+s}, where stack size s ∈ {1, 64, 256}. Audio is 16-bit depth at 16 kHz sample rate.

**Output**: Probability distribution (specifically a 10-component Discrete Mixture of Logistics, DMoL) over the discretized audio values for the stacked frames.

## Scoring recipe

```python
def compute_bpf(predictions, gold, stack_size, num_frames):
    # 1. Compute log-likelihood log p(x) using the model's output distribution (DMoL)
    # 2. For LVMs, use the one-sample ELBO as the log-likelihood estimate
    log_likelihood = compute_log_prob(predictions, gold)
    # 3. Convert to bits per frame: bpf = -log_p_x / (num_frames * log(2))
    bpf = -log_likelihood / (num_frames * np.log(2))
    # 4. Average bpf across all samples in the split
    return np.mean(bpf)
```

## Common pitfalls

- LVMs report the one-sample ELBO, which is a lower bound on the true log-likelihood; the paper marks these with ≤ in tables, meaning the true likelihood could be better (lower bpf).
- Stack size s drastically changes temporal resolution and likelihood values; models must be compared at identical s to avoid unfair comparisons.
- FLAC compression rates are reported on linear PCM, while models are trained on μ-law encoded audio, making direct compression-to-likelihood comparisons approximate.

## Evidence (verbatim from paper)

> We report likelihoods in units of bits per frame (bpf) as this yields a more interpretable and comparable likelihood than total likelihood in nats. It also has direct connections with information theory and compression (Shannon, 1948; Townsend et al., 2019). In units of bits per frame, lower is better. For LVMs, we report the one-sample ELBO.

## Citation

```bibtex
@misc{havtorn2022benchmarking,
  title={Benchmarking Generative Latent Variable Models for Speech},
  author={Havtorn et al. (2022)},
  year={2022},
  note={arXiv:2202.12707}
}
```

- arXiv: 2202.12707