Vocsim Eval

vocsim-eval

VocSim: A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio — Basha et al. (2025) (arXiv:2512.10120, 2025)

What this evaluates

Evaluates the intrinsic geometric alignment and zero-shot content identity of frozen audio embeddings across diverse single-source audio corpora. It measures how well models can retrieve semantically similar audio clips without task-specific fine-tuning, highlighting generalization gaps on low-resource or out-of-distribution speech.

Datasets

• VocSim — total 125000; splits: test (125000); repo https://github.com/anonymoussubmission0000/vocsim

Metrics

• P@1 — range: percent
  • Precision at rank 1; measures the proportion of queries where the top retrieved embedding matches the true class label. Serves as a local retrieval accuracy metric sensitive to dataset structure.
• P@5 — range: percent
  • Precision at rank 5; measures the proportion of correct matches within the top 5 retrieved embeddings. Also a local metric that degrades with increasing class counts.
• GSR (primary) — range: percent
  • Global Separation Rate; a permutation-calibrated metric that measures the proportion of correctly separated embedding boundaries relative to a random label baseline. Calibrated via 300 permutation resamples to account for dataset density and class count, making it robust to OOD distribution shifts.

Input / output format

Input: Single-source audio clips (human speech, animal vocalizations, environmental sounds) processed through frozen audio encoders (e.g., Whisper, CLAP, WavLM, BEATs, EAT) to extract embeddings. Features are optionally aggregated via statistical pooling (mean-time/freq), CLS tokens, or reduced via transductive PCA.

Output: Similarity rankings of audio clips based on embedding distances (Euclidean or Cosine/Spearman), yielding Precision@k and Global Separation Rate scores per subset.

Scoring recipe

def compute_precision_at_k(retrieved_ids, true_ids, k):
    return len(set(retrieved_ids[:k]) & set(true_ids)) / k

def compute_gsr(embeddings, labels, n_perms=300):
    raw_separation = calculate_boundary_separation(embeddings, labels)
    perm_baseline = sum(
        calculate_boundary_separation(embeddings, permute(labels))
        for _ in range(n_perms)
    ) / n_perms
    return raw_separation - perm_baseline

Common pitfalls

• Confusing raw GSR with permutation-calibrated GSR; raw values are heavily inflated by dataset density and class count, requiring permutation baselines for meaningful OOD comparison.
• Assuming transductive PCA improves out-of-distribution performance; ablations show it yields negligible gains on blind sets due to geometric manifold collapse where whitening merely rotates noise.
• Treating local metrics (P@1/P@5) and global metrics (GSR) as interchangeable; P@k degrades sharply on low-resource speech while GSR remains stable, masking the true generalization gap.

Evidence (verbatim from paper)

Table 2: Zero-Shot Content Identity Performance. Values are macro-averages ± margin of error across subsets, derived from per-subset 95% bootstrap confidence intervals (300 resamples). ... Local metrics like P@1 and P@5 (blue lines) are highly sensitive to dataset structure, degrading significantly as the number of classes increases (b) while improving with more samples per class (c). In contrast, the global metric GSR (yellow line) remains remarkably stable across these conditions, suggesting it captures a more intrinsic property of the embedding geometry.

Citation

@misc{basha2025vocsim,
  title={VocSim: A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio},
  author={Basha et al. (2025)},
  year={2025},
  note={arXiv:2512.10120}
}

• arXiv: 2512.10120