Mseb Eval

mseb-eval

Massive Sound Embedding Benchmark (MSEB) — Heigold et al. (2026) (arXiv:2602.07143, 2026)

What this evaluates

Evaluates audio embedding models and ASR pipelines across eight core auditory tasks to measure real-world generalization, compression robustness, and the performance gap between direct audio processing and text-based oracles. It quantifies how well embeddings capture semantic and acoustic information without task-specific fine-tuning.

Datasets

• SVQ (Simple Voice Questions) — total ?; splits: test (-1)
• Speech-MASSIVE — total ?; splits: test (-1)
• FSD50K — total ?; splits: test (-1)
• BirdSet — total ?; splits: test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of the reciprocal of the rank of the first correct item in a ranked list of candidates.
• mAP — range: [0, 1]
  • mean Average Precision: the area under the precision-recall curve, averaging precision across all recall levels for each query.
• F1 — range: [0, 1]
  • F1 score: the harmonic mean of precision and recall, balancing false positives and false negatives.
• WER — range: percent
  • Word Error Rate: the percentage of words incorrectly transcribed relative to the ground truth, calculated as (substitutions + deletions + insertions) / total words.
• NDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain: measures ranking quality by discounting the relevance of items based on their position in the list.
• V-measure — range: [0, 1]
  • V-measure: the harmonic mean of homogeneity and completeness, evaluating clustering quality against ground-truth labels.
• FAD — range: other
  • Fréchet Audio Distance: computes the Fréchet distance between the multivariate Gaussian distributions of features extracted from real and reconstructed audio.

Input / output format

Input: Audio recordings (speech, environmental, or bioacoustic) or ground-truth text transcripts, paired with queries, candidate hypotheses, class labels, or target audio signals.

Output: Embedding vectors, predicted class labels, ranked candidate lists, transcribed text, or reconstructed audio waveforms.

Scoring recipe

def compute_metric(task, predictions, gold):
    if task == 'retrieval':
        return mean([1.0 / rank for rank in predictions])  # MRR
    elif task in ('reranking', 'classification'):
        return average_precision(gold, predictions)  # mAP
    elif task == 'reasoning':
        return f1_score(gold, predictions)  # F1
    elif task == 'transcription':
        return wer(gold, predictions)  # WER
    elif task == 'segmentation':
        return ndcg_at_k(gold, predictions)  # NDCG
    elif task == 'clustering':
        return v_measure_score(gold, predictions)  # V-measure
    elif task == 'reconstruction':
        return frechet_audio_distance(gold, predictions)  # FAD

Common pitfalls

• Assuming ASR quality (WER) perfectly correlates with downstream task performance, as retrieval and reasoning depend disproportionately on semantically salient terms rather than literal word accuracy.
• Treating unbounded metrics like FAD or WER as having a fixed upper bound; the benchmark explicitly notes they are open-ended and visualized with dashed lines.
• Ignoring locale/domain variability; aggregate scores mask extreme performance drops for low-resource languages or noisy conditions, which the paper highlights as a key finding.

Evidence (verbatim from paper)

Retrieval was performed via dot product between the query embedding and document embeddings, with quality measured by Mean Reciprocal Rank (MRR).

Citation

@misc{heigold2026mseb,
  title={Massive Sound Embedding Benchmark (MSEB)},
  author={Heigold et al. (2026)},
  year={2026},
  note={arXiv:2602.07143}
}

• arXiv: 2602.07143