# Syntheory Eval

> Probes whether music foundation models encode discrete and continuous Western music theory concepts by training linear or MLP classifiers on their internal audio embeddings. Use when the user wants to benchmark on SynTheory, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/syntheory-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/syntheory-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/syntheory-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/syntheory-eval

---


# syntheory-eval

> Do Music Generation Models Encode Music Theory? — Wei et al. (2024) (arXiv:2410.00872, 2024)

## What this evaluates

Probes whether music foundation models encode discrete and continuous Western music theory concepts by training linear or MLP classifiers on their internal audio embeddings.

## Datasets

- **SynTheory** — total ?; splits: train (-1), test (-1), val (-1); repo https://github.com/brown-palm/syntheory

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels out of total predictions for classification tasks (notes, intervals, scales, chords, chord progressions, time signatures).
- `R^2` — range: other
  - Coefficient of determination for the continuous tempo regression task, calculated as 1 - (SS_res / SS_tot) where SS is sum of squared errors.

## Input / output format

**Input**: Mean-pooled audio embeddings (dimension varies by model and layer) extracted from 4-second mono audio clips.

**Output**: Predicted class label (for classification) or predicted BPM value (for regression).

## Scoring recipe

```python
def score(predictions, gold, task_type):
    if task_type == "tempo":
        ss_res = sum((y_true - y_pred)**2 for y_true, y_pred in zip(gold, predictions))
        ss_tot = sum((y_true - mean(gold))**2 for y_true in gold)
        return 1 - (ss_res / ss_tot)
    else:
        correct = sum(1 for y_true, y_pred in zip(gold, predictions) if y_true == y_pred)
        return correct / len(gold)
```

## Common pitfalls

- Mean-pooling across time discards temporal dynamics, which may unfairly penalize models on time-varying concepts like chord progressions.
- The tempo split stratifies by BPM range (middle 70% train, extremes test/val) to test generalization, which does not reflect a uniform BPM distribution in real-world music.
- Layer selection is performed independently per concept and model using the validation set, introducing potential selection bias if not properly cross-validated.

## Evidence (verbatim from paper)

> For the classification tasks, we measure the accuracy of our trained probes on the following SynTheory tasks: Notes (12): C, C#, D, D#, E, F, F#, G, G#, A, A#, and B - Intervals (12)... Time Signatures (8)... These tasks are trained on a 70% train, 15% test, and 15% validation split... For the Tempo dataset, we train a regression probe, over the 161 tempo values... We use MSE loss and report the R^2 score.

## Citation

```bibtex
@misc{wei2024do,
  title={Do Music Generation Models Encode Music Theory?},
  author={Wei et al. (2024)},
  year={2024},
  note={arXiv:2410.00872}
}
```

- arXiv: 2410.00872