World Music Corpora Eval

world-music-corpora-eval

Universal Music Representations? Evaluating Foundation Models on World Music Corpora — Papaioannou et al. (2025) (arXiv:2506.17055, 2025)

What this evaluates

Evaluates audio foundation models' cross-cultural generalization across diverse musical traditions (Western, Greek, Turkish, Indian) using multi-label tagging and few-shot learning. Probes whether pre-trained representations capture cultural musical knowledge without extensive adaptation.

Datasets

• Turkish-makam — total ?; splits: train (-1), val (-1), test (-1)
• Hindustani — total ?; splits: train (-1), val (-1), test (-1)
• Carnatic — total ?; splits: train (-1), val (-1), test (-1)
• MagnaTagATune — total ?; splits: train (-1), val (-1), test (-1)
• FMA-medium — total ?; splits: train (-1), val (-1), test (-1)
• Lyra — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the model's ability to distinguish between positive and negative labels across all classification thresholds, averaged across all multi-label tags.
• mAP — range: [0, 1]
  • Mean Average Precision. Computes the average precision for each label and averages them across all tags. Suitable for multi-label classification where label distributions are imbalanced.
• Macro-F1 — range: [0, 1]
  • Harmonic mean of precision and recall calculated per class, then averaged equally across all classes. Gives equal weight to rare and frequent tags.
• Micro-F1 — range: [0, 1]
  • Harmonic mean of global precision and recall calculated across all instances and classes. Accounts for class imbalance by weighting metrics by support size.

Input / output format

Input: Mono audio clips processed in model-specific windows (10–30 seconds) and resampled to model-specific rates (16–48 kHz), paired with multi-label genre/tradition tags.

Output: Per-class prediction scores or probabilities for multi-label classification; for few-shot learning, discrete class predictions based on cosine distance to support set prototypes.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score, f1_score

def compute_metrics(y_true, y_scores, task='probing'):
    if task in ['probing', 'sft']:
        auc = roc_auc_score(y_true, y_scores, average='macro')
        map_score = average_precision_score(y_true, y_scores, average='macro')
        return {'ROC-AUC': auc, 'mAP': map_score}
    else: # ML-FSL
        y_pred = (y_scores >= 0.5).astype(int)
        macro_f1 = f1_score(y_true, y_pred, average='macro')
        micro_f1 = f1_score(y_true, y_pred, average='micro')
        return {'Macro-F1': macro_f1, 'Micro-F1': micro_f1}

Common pitfalls

• Data leakage if ML-FSL evaluation samples are not strictly drawn from held-out test sets.
• Model-specific audio preprocessing (window length, sampling rate) must be strictly followed to avoid representation mismatch.
• Standard accuracy is inappropriate; multi-label metrics (ROC-AUC, mAP, F1 variants) must be used due to overlapping genre tags.

Evidence (verbatim from paper)

For the Probing and SFT methodologies, we report area under the receiver operating characteristic curve (ROC-AUC) and mean average precision (mAP). These metrics are particularly well-suited for multi-label classification tasks [51] and are consistent with prior work in music tagging [17, 28]. For ML-FSL evaluation, we report macro-F1 (M-F1) and micro-F1 (mF1) scores, which align with the LC-Protonets evaluation framework [29]. F1 score is the harmonic mean of the precision and recall scores. Macro-F1 gives equal weight to all classes, while micro-F1 accounts for class imbalance by calculating metrics globally across all instances.

Citation

@misc{papaioannou2025universal,
  title={Universal Music Representations? Evaluating Foundation Models on World Music Corpora},
  author={Papaioannou et al. (2025)},
  year={2025},
  note={arXiv:2506.17055}
}

• arXiv: 2506.17055