Duo Tok Eval

duo-tok-eval

DUO-TOK: Dual-Track Semantic Music Tokenizer for Vocal-Accompaniment Generation — Rui Lin et al. (2025) (arXiv:2511.20224, 2025)

What this evaluates

Evaluates semantic music tokenizers on their ability to preserve musical semantics for conditional generation, their efficiency for language modeling, and their audio reconstruction fidelity. It probes whether decoupled vocal-accompaniment tokenization yields better LM-friendliness and tagging performance without sacrificing perceptual quality.

Datasets

• MagnaTagATune — total ?; splits: test (-1)

Metrics

• PPL@1024 (primary) — range: other
  • Vocabulary-normalized perplexity. Normalizes the model's cross-entropy loss H to an equivalent 1,024-way prediction problem: PPL@1024 = 2^(H * log2(1024) / log2(S)), where S is the tokenizer's vocabulary size. Allows fair comparison across tokenizers with different codebook sizes.
• MTT AP — range: [0, 1]
  • Average Precision for multi-label music tagging (genre, instrumentation, mood). Computed by training a linear classifier on frozen tokenizer representations and averaging precision across tags.
• Top-k accuracy — range: [0, 1]
  • Fraction of next-token predictions where the ground truth token appears in the top-k most likely predictions. Reported for k=1, 5, 10, 50.
• PESQ — range: other
  • Perceptual Evaluation of Speech Quality. Standard ITU-T P.862 metric measuring perceptual audio quality between original and reconstructed signals.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility. Measures speech/audio intelligibility based on temporal envelope correlation.
• Mel L1 — range: other
  • L1 distance between log-Mel spectrograms of the original and reconstructed audio.

Input / output format

Input: Frozen tokenizer representations (for MTT classification); discrete token sequences (for LM tasks); original and reconstructed audio waveforms (for reconstruction metrics).

Output: Multi-label tag probabilities; next-token predictions; reconstructed audio waveforms.

Scoring recipe

def compute_ppl_1024(cross_entropy_loss, vocab_size):
    normalized_h = cross_entropy_loss * (math.log2(1024) / math.log2(vocab_size))
    return 2 ** normalized_h

def top_k_accuracy(predictions, targets, k):
    top_k_indices = torch.topk(predictions, k, dim=1).indices
    correct = top_k_indices.eq(targets.view(-1, 1)).any(dim=1)
    return correct.float().mean()

Common pitfalls

• Raw top-k accuracy is heavily biased by vocabulary size; small vocabularies get a numerical advantage. Always report vocabulary-normalized PPL@1024 alongside top-k.
• Reconstruction metrics must be compared at similar bitrates (kbps) to avoid conflating compression ratio with fidelity.
• Pseudo-stems generated by Demucs may contain separation artifacts that affect reconstruction and tagging metrics compared to true multi-track data.

Evidence (verbatim from paper)

Following AudioCodecBench and recent music codec work, we evaluate tokenizers along three axes: music tagging performance, LM-friendliness, and reconstruction quality. To factor out this trivial difficulty gap, following AudioCodecBench we normalize the LM cross-entropy H to an equivalent 1,024-way prediction problem and define PPL@1024 (the formal definition is given in Section 4.5). We always report the actual vocabulary size S and top-k accuracy alongside PPL@1024, but use PPL@1024 as the primary axis when comparing intrinsic modeling difficulty across tokenizers, so that improvements do not come merely from shrinking the label space. For reconstruction, we report perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), and the L1 distance between log-Mel spectrograms.

Citation

@misc{lin2025duotok,
  title={DUO-TOK: Dual-Track Semantic Music Tokenizer for Vocal-Accompaniment Generation},
  author={Rui Lin et al. (2025)},
  year={2025},
  note={arXiv:2511.20224}
}

• arXiv: 2511.20224