Cnn Music Tagging Eval

cnn-music-tagging-eval

Evaluation of CNN-based Automatic Music Tagging Models — Won et al. (2020) (arXiv:2006.00751, 2020)

What this evaluates

This evaluation benchmarks CNN-based models on automatic music tagging, measuring their ability to predict multiple genre, instrument, and mood labels from audio spectrograms. It assesses both standard classification performance and robustness to audio transformations like time-stretching and pitch shifting.

Datasets

• MagnaTagATune — total ?; splits: test (-1)
• Million Song Dataset — total ?; splits: test (-1)
• MTG-Jamendo — total ?; splits: test (-1)

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, plotting the true positive rate against the false positive rate across all classification thresholds.
• PR-AUC — range: [0, 1]
  • Area under the Precision-Recall curve, summarizing the trade-off between precision and recall across thresholds. It is particularly sensitive to class imbalance in multi-label settings.

Input / output format

Input: Audio tracks or short excerpts (~3.69s–5s) converted to Mel spectrograms (typically 96 or 128 frequency bands).

Output: Binary classification probabilities or scores for each of the top 50 tags per dataset.

Scoring recipe

def compute_tag_auc(y_true, y_pred, metric='roc'):
    tag_scores = []
    for t in range(y_true.shape[1]):
        if metric == 'roc':
            tag_scores.append(roc_auc_score(y_true[:, t], y_pred[:, t]))
        else:
            tag_scores.append(average_precision_score(y_true[:, t], y_pred[:, t]))
    return np.mean(tag_scores)

# Macro-average across all tags for each dataset
roc_auc = compute_tag_auc(gold_labels, predictions, 'roc')
pr_auc = compute_tag_auc(gold_labels, predictions, 'pr')

Common pitfalls

• Comparing models trained with different Mel band counts (e.g., 96 vs 128) without re-tuning hyperparameters can unfairly penalize architectures originally optimized for fewer bands.
• Training on short audio chunks introduces label noise, as a tag may be valid for the full track but absent in the specific excerpt, complicating performance interpretation.
• Relying solely on ROC-AUC can be misleading for highly imbalanced multi-label tagging tasks; PR-AUC is often a more reliable indicator of practical utility.

Evidence (verbatim from paper)

We report ROC-AUC and PR-AUC of all implemented models using three datasets in Table 2. In general, models trained with short audio excerpts (Musicnn, variants of sample-level CNN, Self-attention, Harmonic CNN, variants of short-chunk CNN) outperform other models trained with relatively longer audio segments (FCN, CRNN).

Citation

@misc{won2020evaluation,
  title={Evaluation of CNN-based Automatic Music Tagging Models},
  author={Won et al. (2020)},
  year={2020},
  note={arXiv:2006.00751}
}

• arXiv: 2006.00751