Music Tagging Eval

music-tagging-eval

Toward Interpretable Music Tagging with Self-Attention — Won et al. (2019) (arXiv:1906.04972, 2019)

What this evaluates

Evaluates a model's ability to predict multiple audio tags (e.g., genre, mood, instruments) from short audio segments. It probes long-range temporal dependency modeling and robustness to class imbalance in user-generated music metadata.

Datasets

• MagnaTagATune (MTAT) — total ?; splits: test (-1)
• Million Song Dataset (MSD) — total ?; splits: test (-1)

Metrics

• AUPR (primary) — range: [0, 1]
  • Area Under the Precision-Recall curve. Computed by integrating precision over recall thresholds across all predicted probabilities for each tag. Preferred for highly skewed, multi-label datasets.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across classification thresholds.

Input / output format

Input: Log mel-spectrograms of audio segments (typically ~4.1s or ~16.4s long).

Output: Multi-label binary predictions (probabilities) for a fixed vocabulary of music tags (e.g., genre, mood, instrument).

Scoring recipe

def compute_aupr(y_true, y_pred):
    precisions, recalls, _ = precision_recall_curve(y_true, y_pred)
    return auc(recalls, precisions)

def compute_auroc(y_true, y_pred):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

# Macro-average across all tags
aupr = np.mean([compute_aupr(y_true[:, i], y_pred[:, i]) for i in range(num_tags)])
auroc = np.mean([compute_auroc(y_true[:, i], y_pred[:, i]) for i in range(num_tags)])

Common pitfalls

• AUROC can be misleading on these datasets due to high popularity bias and class skewness; AUPR is explicitly recommended instead.
• Attention heatmaps indicate where the model focuses but do not explain classification reasoning, potentially highlighting irrelevant loud segments for quiet tags.
• Model performance drops when input sequence length increases from ~4.1s to ~16.4s without adjusting architecture depth or receptive field.

Evidence (verbatim from paper)

Following previous research [[28]], we report the Area Under Precision Recall curve (AUPR) along with conventional Area Under Receiver Operating Characteristic curve (AUROC). AUPR is known to be more informative to evaluate the algorithm’s performance when it deals with highly skewed datasets [[7]]. Since we are using user-generated tags (MTAT and MSD), there is popularity biased skewness in their distributions.

Citation

@misc{won2019toward,
  title={Toward Interpretable Music Tagging with Self-Attention},
  author={Won et al. (2019)},
  year={2019},
  note={arXiv:1906.04972}
}

• arXiv: 1906.04972