Maeb Eval

maeb-eval

MAEB: Massive Audio Embedding Benchmark — El Assadi et al. (2026) (arXiv:2602.16008, 2026)

What this evaluates

Evaluates audio embedding models across 30 tasks spanning speech, music, environmental sounds, bioacoustics, emotion recognition, and cross-modal audio-text reasoning in over 100 languages. It probes the ability of models to generalize across acoustic domains, handle multilingual alignment, and perform both supervised and unsupervised audio understanding tasks.

Datasets

• MAEB — total ?; splits: test (-1)

Metrics

• Average Score (primary) — range: percent
  • Computed as the arithmetic mean of task-specific scores (accuracy for classification, similarity/retrieval scores for retrieval and reranking, and clustering metrics) across all 30 tasks. Reported as a percentage.
• Borda Count — range: other
  • A ranking aggregation method where models receive points based on their rank per task; the model with the highest total points ranks first overall.

Input / output format

Input: Raw audio clips (and text prompts for cross-modal tasks).

Output: Fixed-dimensional embeddings used for downstream task evaluation (classification, retrieval, clustering, reranking).

Scoring recipe

task_scores = []
for task in MAEB_TASKS:
    preds = model.predict(task.inputs)
    gold = task.labels
    score = task.metric(preds, gold)
    task_scores.append(score)
return sum(task_scores) / len(task_scores)

Common pitfalls

• Assuming a single model dominates all acoustic domains; the benchmark reveals strong specialization where speech, music, and environmental models excel in disjoint areas.
• Cross-modal retrieval performance collapses in multilingual settings despite good audio-only performance, indicating a critical gap in cross-lingual audio-text alignment.
• Clustering tasks are universally challenging and poorly correlated with supervised task performance, revealing a fundamental gap in semantic structure of current embeddings.

Evidence (verbatim from paper)

Table 2 presents the top 30 models on the MAEB benchmark. The table includes both MAEB rank (over all 30 tasks) and Audio-only rank (over the 19 audio-only subset tasks) to highlight how models perform differently across task types. LCO-Embedding-Omni-7B ranks first overall by Borda count, achieving the highest average scores (52.2% overall, 50.3% cross-modal retrieval, 64.5% zero-shot) across all categories.

Citation

@misc{elassadi2026maeb,
  title={MAEB: Massive Audio Embedding Benchmark},
  author={El Assadi et al. (2026)},
  year={2026},
  note={arXiv:2602.16008}
}

• arXiv: 2602.16008