Musictheorybench Eval

musictheorybench-eval

ChatMusician: Understanding and Generating Music Intrinsically with LLM — Yuan et al. (2024) (arXiv:2402.16153, 2024)

What this evaluates

Evaluates a model's ability to reason about music theory concepts and understand symbolic music representations, alongside general language knowledge and structured music generation capabilities.

Datasets

• MusicTheoryBench — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)

Metrics

• average accuracy (primary) — range: [0, 1]
  • Mean accuracy across all questions, computed by averaging results over five different random shuffles of the multiple-choice options to mitigate position bias.
• parsing success rate — range: [0, 1]
  • Percentage of generated music sequences that successfully parse into valid ABC notation structure.
• phrase-level repetition metric — range: other
  • Measures the frequency of repeated musical phrases within generated compositions.
• average percentile score — range: [0, 1]
  • Gauges model controllability by comparing generated outputs against a baseline distribution to compute percentile rankings.

Input / output format

Input: For understanding tasks: multiple-choice questions in text format (zero-shot for MusicTheoryBench, 5-shot for MMLU). For generation tasks: text prompts requesting musical composition.

Output: For understanding tasks: the selected option letter or answer string. For generation tasks: ABC notation representing the generated music score.

Scoring recipe

# For MusicTheoryBench accuracy
accuracies = []
for shuffle in range(5):
    shuffled_options = shuffle_options(gold_options)
    preds = model.predict(input, options=shuffled_options)
    correct = sum(1 for p, g in zip(preds, gold_labels) if p == g)
    accuracies.append(correct / len(gold_labels))
final_accuracy = sum(accuracies) / 5

# For parsing success rate
valid_parses = sum(1 for gen in generated_music if parse_abc(gen) is not None)
parsing_success_rate = valid_parses / len(generated_music)

Common pitfalls

• Option shuffling must be applied before averaging accuracy to avoid position bias artifacts.
• Parsing success rate only validates syntactic format, not musical correctness or quality.
• Human judgment is required for musicality, so automated metrics alone are insufficient for full evaluation.

Evidence (verbatim from paper)

For the MusicTheoryBench, we report the average accuracy after shuffling the option five times as the final results under a zero-shot setting. Additionally, we have developed two specific metrics: a phrase-level repetition metric and a parsing success rate metric, aimed at assessing the structuration and format accuracy of the generated music. Furthermore, we introduce an average percentile score metric to gauge the models' controllability.

Citation

@misc{yuan2024chatmusician,
  title={ChatMusician: Understanding and Generating Music Intrinsically with LLM},
  author={Yuan et al. (2024)},
  year={2024},
  note={arXiv:2402.16153}
}

• arXiv: 2402.16153