qhjqhj00 ssmr-bench-eval
Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning — Wang et al. (2025) (arXiv:2509.04059, 2025)
What this evaluates
Evaluates musical reasoning capabilities across rhythm, chords, intervals, and scales using sheet music problems. It tests both textual and visual (staff notation) modalities to measure how well models recognize musical elements and perform logical deductions.
Datasets
- Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench) — total 3200; splits: test (3200)
Metrics
accuracy(primary) — range: [0, 100] percent- Calculated as the percentage of correctly answered multiple-choice questions out of the total number of questions. Overall accuracy is the average across all sub-tasks (Rhythm, Chord, Interval, Scale).
Input / output format
Input: Multiple-choice questions about sheet music properties, provided either as textual descriptions or as images of musical staff notation. Model options are shuffled before inference.
Output: Free-text model response containing the selected option or reasoning, which is subsequently parsed to extract the final answer.
Scoring recipe
def compute_accuracy(predictions, golds):
correct = sum(1 for p, g in zip(predictions, golds) if p.strip().upper() == g.strip().upper())
return (correct / len(golds)) * 100
Common pitfalls
- Options are shuffled to prevent models from exploiting positional bias or contamination.
- Answers must be parsed using the Math-Verify tool rather than simple string matching.
- Visual modality performance is heavily bottlenecked by the model's ability to first recognize musical symbols before reasoning.
Evidence (verbatim from paper)
Specifically, GPT-5 attains the highest overall accuracy at 97.00%, followed by Gemini-2.5-Pro at 96.10%, DeepSeek-R1 at 93.63%, and GPT-4.1 at 79.81%.
Citation
@misc{wang2025towards,
title={Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning},
author={Wang et al. (2025)},
year={2025},
note={arXiv:2509.04059}
}
- arXiv: 2509.04059