Music Audio Representation Eval

music-audio-representation-eval

Supervised and Unsupervised Learning of Audio Representations for Music Understanding — McCallum et al. (2022) (arXiv:2210.03799, 2022)

What this evaluates

Evaluates the quality of pre-trained audio embeddings for downstream music understanding tasks including tagging, genre classification, mood prediction, pitch/instrument detection, key classification, and emotion recognition. It tests whether frozen embeddings can be effectively probed with simple MLP classifiers to achieve competitive performance without fine-tuning the backbone model.

Datasets

• MSDS — total 242000; splits: published (-1); repo https://github.com/minzwon/tag-based-music-retrieval
• MSD50 — total 36000; splits: published (-1); repo https://github.com/minzwon/tag-based-music-retrieval
• MSD100 — total 115000; splits: published (-1); repo https://github.com/minzwon/tag-based-music-retrieval
• MSD500 — total 156000; splits: published (-1); repo https://github.com/minzwon/tag-based-music-retrieval
• AMM — total 67000; splits: artist-based (-1); repo https://github.com/fdlm/listening-moods
• MuMu — total 147000; splits: official (-1); repo https://zenodo.org/record/1236906#.YoPIAhNBx0s
• MTT — total 26000; splits: published (-1); repo https://github.com/jongpillee/music_dataset_split
• NSynthP — total 306000; splits: published (-1)
• NSynthI — total 306000; splits: published (-1)
• GTZAN — total 930; splits: fault-filtered (-1)
• Emo — total 744; splits: artist-based (-1)
• GSKey — total 2100; splits: train/val/test (-1); repo https://github.com/GiantSteps/giantsteps-mtg-key-dataset
• Jam-50 — total 54000; splits: official (-1); repo https://github.com/MTG/mtg-jamendo-dataset
• Jam-All — total 56000; splits: official (-1); repo https://github.com/MTG/mtg-jamendo-dataset
• Jam-MT — total 18000; splits: official (-1); repo https://github.com/MTG/mtg-jamendo-dataset

Metrics

• weighted accuracy (primary) — range: [0, 1]
  • Standard classification accuracy weighted by class frequency in the test set, commonly used in music key detection benchmarks.
• coefficient of determination (R²) — range: other
  • R² = 1 - (SS_res / SS_tot), where SS_res is the sum of squared residuals between predictions and ground truth, and SS_tot is the total sum of squares. Used for continuous emotion valence and arousal regression.

Input / output format

Input: Timeline-averaged global average pooled activations from the final convolutional layer of the SF-NFNet-F0 architecture, sampled at 0.5 Hz from 3-second audio snippets. These fixed embeddings are fed into a simple multi-layer perceptron (MLP) classifier.

Output: Discrete class labels for classification tasks (tagging, genre, mood, pitch, instrument, key), or continuous scalar values for regression tasks (emotion valence and arousal).

Scoring recipe

def compute_metric(task, preds, golds):
    if task == 'key_classification':
        weights = get_class_frequencies(golds)
        correct = sum(w * (p == g) for w, p, g in zip(weights, preds, golds))
        return correct / sum(weights)
    elif task == 'emotion_regression':
        ss_res = sum((p - g)**2 for p, g in zip(preds, golds))
        ss_tot = sum((g - mean(golds))**2 for g in golds)
        return 1 - (ss_res / ss_tot)
    else:
        return sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)

Common pitfalls

• Failing to use artist-based splits for datasets like AMM and Emo, which causes severe data leakage and inflated performance due to artist correlation in the labels.
• Over-optimizing probe hyperparameters without constraining them to prior SOTA ranges, which confounds embedding quality evaluation with probe tuning capabilities.
• Ignoring dataset-specific windowing strategies (e.g., 1-second vs 4-second feature windows for NSynth), which drastically changes embedding aggregation and downstream probe performance.

Evidence (verbatim from paper)

In all cases, we train probes using Adam optimization with a cosine learning rate schedule with 1,000 steps of warmup followed by a decay to zero over the remainder of the steps. We optimize the learning rate, number of training steps, and l2-regularization of each probe to achieve best performance / prevent overfitting. For evaluation we compute a weighted accuracy score common in key classification... This poses a regression problem—we use the coefficient of determination as the evaluation metric for both valence (EmoV) and arousal (EmoA).

Citation

@misc{mccallum2022music,
  title={Supervised and Unsupervised Learning of Audio Representations for Music Understanding},
  author={McCallum et al. (2022)},
  year={2022},
  note={arXiv:2210.03799}
}

• arXiv: 2210.03799