Csymr Eval

csymr-eval

CSyMR: Benchmarking Compositional Symbolic Muisc Reasoning With MIR Tool Integration — Wang et al. (2025) (arXiv:2601.11556, 2025)

What this evaluates

Evaluates compositional symbolic music reasoning by requiring models to chain atomic analyses across multiple musical dimensions (e.g., rhythm, harmony, key, structure) to answer multiple-choice questions derived from expert forums and professional exams.

Datasets

• CSyMR-Bench — total 126; splits: test (126)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions out of the total number of questions.

Input / output format

Input: Symbolic score excerpts, a natural-language question, and four answer options.

Output: Answer letter selection (A, B, C, or D) for direct prompting; ReAct-style tool invocation steps and final answer for the tool-augmented agent.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / len(predictions)) * 100

Common pitfalls

• Models may hallucinate symbolic music details (e.g., non-existent pitches or chords) when reasoning without tool augmentation, leading to incorrect harmonic analysis.
• Tool-augmented methods may underperform on metadata- or style-driven categories if the model's reasoning focus shifts away from leveraging the analysis tools effectively.

Evidence (verbatim from paper)

Table[1] summarizes per‐category results across prompting and reasoning methods. Few‐shot prompting and explicit reasoning improve over direct prompting, but ReAct (without musical tools) is only comparable to CoT. Our method achieves the best overall accuracy, consistently outperforming all baselines.

Citation

@misc{wang2025csymr,
  title={CSyMR: Benchmarking Compositional Symbolic Muisc Reasoning With MIR Tool Integration},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2601.11556}
}

• arXiv: 2601.11556