Music Autotagging Eval

music-autotagging-eval

Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets — Ramoneda et al. (2025) (arXiv:2509.06936, 2025)

What this evaluates

Evaluates audio representation models on music autotagging tasks, measuring how well they predict categorical genre/instrument/mood tags and continuous musical features from audio input. It compares performance across generic tag datasets and expert-annotated continuous features to highlight limitations in current evaluation practices.

Datasets

• MagnaTagATune — total ?; splits: test (-1)
• MTG-Jamendo — total ?; splits: test (-1)
• MGPHot-tag — total ?; splits: test (-1)
• MGPHot-reg — total ?; splits: test (-1)

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision: computes the average precision across all tags/classes by averaging precision at each recall level for each class, then averaging across classes.
• RMSE — range: other
  • Root Mean-Squared Error: measures the square root of the mean squared difference between predicted and ground-truth continuous values.

Input / output format

Input: Raw audio waveform or precomputed audio embeddings.

Output: Categorical tag probabilities/labels for classification tasks; continuous numerical values for regression tasks.

Scoring recipe

def compute_map(preds, labels):
    ap = []
    for c in range(preds.shape[1]):
        scores = preds[:, c]
        truth = labels[:, c]
        sorted_idx = np.argsort(-scores)
        tp = np.cumsum(truth[sorted_idx])
        fp = np.cumsum(1 - truth[sorted_idx])
        prec = tp / (tp + fp + 1e-8)
        ap.append(np.mean(prec[truth[sorted_idx] == 1]))
    return np.mean(ap)

def compute_rmse(preds, labels):
    return np.sqrt(np.mean((preds - labels) ** 2))

# Average over 5 random seeds
final_map = np.mean([compute_map(p, l) for p, l in runs])
final_rmse = np.mean([compute_rmse(p, l) for p, l in runs])

Common pitfalls

• Using official MTG-Jamendo splits instead of the paper's custom split that includes all available tags per category.
• Assuming category names (e.g., 'Genre', 'Instrument') are directly comparable across datasets, as underlying tag definitions differ substantially.
• Reporting MAE/MSE instead of the primary RMSE metric without noting the paper's explicit choice of RMSE for interpretability.

Evidence (verbatim from paper)

Table[3] reports the mean average precision (MAP ↑) for the three tagging tasks and root mean-squared error (RMSE ↓) for the regression task. Each score is the mean of five runs initialized with different seeds.

Citation

@misc{ramoneda2025mgphot,
  title={Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets},
  author={Ramoneda et al. (2025)},
  year={2025},
  note={arXiv:2509.06936}
}

• arXiv: 2509.06936