Syntheory Eval

syntheory-eval

Do Music Generation Models Encode Music Theory? — Wei et al. (2024) (arXiv:2410.00872, 2024)

What this evaluates

Probes whether music foundation models encode discrete and continuous Western music theory concepts by training linear or MLP classifiers on their internal audio embeddings.

Datasets

• SynTheory — total ?; splits: train (-1), test (-1), val (-1); repo https://github.com/brown-palm/syntheory

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels out of total predictions for classification tasks (notes, intervals, scales, chords, chord progressions, time signatures).
• R^2 — range: other
  • Coefficient of determination for the continuous tempo regression task, calculated as 1 - (SS_res / SS_tot) where SS is sum of squared errors.

Input / output format

Input: Mean-pooled audio embeddings (dimension varies by model and layer) extracted from 4-second mono audio clips.

Output: Predicted class label (for classification) or predicted BPM value (for regression).

Scoring recipe

def score(predictions, gold, task_type):
    if task_type == "tempo":
        ss_res = sum((y_true - y_pred)**2 for y_true, y_pred in zip(gold, predictions))
        ss_tot = sum((y_true - mean(gold))**2 for y_true in gold)
        return 1 - (ss_res / ss_tot)
    else:
        correct = sum(1 for y_true, y_pred in zip(gold, predictions) if y_true == y_pred)
        return correct / len(gold)

Common pitfalls

• Mean-pooling across time discards temporal dynamics, which may unfairly penalize models on time-varying concepts like chord progressions.
• The tempo split stratifies by BPM range (middle 70% train, extremes test/val) to test generalization, which does not reflect a uniform BPM distribution in real-world music.
• Layer selection is performed independently per concept and model using the validation set, introducing potential selection bias if not properly cross-validated.

Evidence (verbatim from paper)

For the classification tasks, we measure the accuracy of our trained probes on the following SynTheory tasks: Notes (12): C, C#, D, D#, E, F, F#, G, G#, A, A#, and B - Intervals (12)... Time Signatures (8)... These tasks are trained on a 70% train, 15% test, and 15% validation split... For the Tempo dataset, we train a regression probe, over the 161 tempo values... We use MSE loss and report the R^2 score.

Citation

@misc{wei2024do,
  title={Do Music Generation Models Encode Music Theory?},
  author={Wei et al. (2024)},
  year={2024},
  note={arXiv:2410.00872}
}

• arXiv: 2410.00872