qhjqhj00 music-audio-tagging-eval
End-to-end learning for music audio tagging at scale — Jordi Pons et al. (2017) (arXiv:1711.02520, 2017)
What this evaluates
This evaluation probes a model's ability to perform large-scale music audio tagging by predicting a fixed set of semantic labels (e.g., genre, mood, instrumentation) from raw 30-second audio clips. It measures how well architectures generalize across varying dataset sizes and label granularities.
Datasets
- MagnaTagATune (MTT) — total 26000; splits: test (-1)
- Million Song Dataset (MSD) — total 240000; splits: test (-1)
- Private Dataset — total 1200000; splits: train (1000000), val (100000), test (100000)
Metrics
top-50 tag prediction(primary) — range: [0, 1]- Multi-label classification metric where the model outputs a ranked list of tags; performance is measured by how many of the ground-truth tags appear in the top-50 predictions.
Input / output format
Input: 30-second music audio clips (waveforms or spectrograms)
Output: Ranked list of top-50 predicted tags per audio clip
Scoring recipe
def score(predictions, gold_tags):
correct = 0
for pred, gold in zip(predictions, gold_tags):
if set(gold).issubset(set(pred)):
correct += 1
return correct / len(predictions)
Common pitfalls
- Artist leakage: Splits are artist-filtered, so models must not memorize artist-specific styles to generalize.
- Dataset scale disparity: Performance trends differ drastically between ~26k samples (MTT) and ~1.2M samples (private), requiring careful scaling analysis.
- Tag granularity: The private dataset uses 139 fine-grained tags (e.g., subgenres, rhythmic feel), which are harder to predict than standard genre tags.
Evidence (verbatim from paper)
Predicting the top-50 tags of this dataset is a popular benchmark for auto-tagging.
Citation
@misc{pons2017endtoend,
title={End-to-end learning for music audio tagging at scale},
author={Jordi Pons et al. (2017)},
year={2017},
note={arXiv:1711.02520}
}
- arXiv: 1711.02520