Discrete Audio Tokenizer Eval

discrete-audio-tokenizer-eval

Discrete Audio Tokens: More Than a Survey! — Mousavi et al. (2025) (arXiv:2506.10274, 2025)

What this evaluates

This benchmark evaluates discrete audio tokenizers across speech, music, and general audio domains. It probes their ability to preserve acoustic fidelity during compression and decompression, as well as their effectiveness when used as inputs for downstream discriminative and generative audio tasks.

Datasets

• LibriSpeech test-clean — total ?; splits: test (-1)
• MUSDB — total ?; splits: test (-1)
• Audioset test-set — total ?; splits: test (-1)
• DASB Benchmark — total ?; splits: test (-1)

Metrics

• SDR (primary) — range: (-inf, inf)
  • Signal-to-Distortion Ratio, measuring the ratio of signal power to distortion power between original and resynthesized audio.
• WER — range: [0, inf)
  • Word Error Rate, calculated as the percentage of words incorrectly recognized compared to the ground truth transcription.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances in discriminative downstream tasks.
• SI-SNR — range: (-inf, inf)
  • Scale-invariant Signal-to-Noise Ratio, invariant to scaling of the reference signal, used for reconstruction quality.

Input / output format

Input: Raw audio waveform (sample rate varies by tokenizer configuration, typically 16kHz or 44.1kHz)

Output: Discrete token indices (codebook IDs) representing the compressed audio representation

Scoring recipe

# Reconstruction Evaluation
tokens = tokenizer.encode(audio)
resynthesized = tokenizer.decode(tokens)
sdr = compute_sdr(audio, resynthesized)
si_snr = compute_si_snr(audio, resynthesized)
wer = compute_wer(asr_model(resynthesized), gold_transcript)

# Downstream Evaluation (DASB)
embeddings = tokenizer.encode(audio)
embeddings = project_to_1024(embeddings) # weighted sum or concatenation
pred = downstream_head(embeddings)
acc = compute_accuracy(pred, gold_labels)

Common pitfalls

• Relying solely on time-domain metrics like SDR/SI-SNR can mislead evaluation, as perceptually optimized tokenizers may score poorly on these metrics despite producing subjectively high-quality audio.
• Evaluating tokenizers only on reconstruction quality ignores their actual utility; task-relevant information preservation must be measured via downstream discriminative and generative benchmarks.
• Domain mismatch: applying speech-trained tokenizers to music or general audio without accounting for training domain differences leads to unfair performance degradation.

Evidence (verbatim from paper)

We assess the quality of resynthesized audio using the original decoder trained for each tokenizer, following protocols from CodecSUPERB and VERSA. We also evaluate the computational efficiency of each tokenizer based on model size (parameters), frame rate, token rate, and multiply-accumulate operations (MACs). ... SDR and SI-SNR are less reliable indicators. A possible reason is that the signal is over-compressed, the generation of neural codec (especially in low-bitrate), usually have less consistency in the local sample-level information.

Citation

@misc{mousavi2025discrete,
  title={Discrete Audio Tokens: More Than a Survey!},
  author={Mousavi et al. (2025)},
  year={2025},
  note={arXiv:2506.10274}
}

• arXiv: 2506.10274