Msu Bench Eval

msu-bench-eval

Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores — Dai et al. (2025) (arXiv:2511.20697, 2025)

What this evaluates

Evaluates large language and vision-language models' ability to comprehend complete musical scores across four hierarchical levels of reasoning. It probes bar localization, structural understanding, and complex musical reasoning, while highlighting modality gaps between textual (ABC notation) and visual (PDF/image) inputs.

Datasets

• MSU-Bench — total 1800; splits: train (90), val (30), test (30)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions. Reported per level (1-4) and overall (aggregate).
• Level-wise Success Rate (LSR) — range: [0, 1]
  • LSR(l) = Correct(Q_{1:l}) / |Q_{1:l}|, where Correct(Q_{1:l}) is the number of scores where all questions from Level 1 through l are answered correctly, and |Q_{1:l}| is the total number of scores. Evaluated with 95% Wilson score confidence intervals.

Input / output format

Input: Textual QA: ABC notation representation of the musical score paired with a question. Visual QA: PDF document/image of the complete musical score paired with a question.

Output: Free-form text answer to the question. Correctness is determined via a majority-vote LLM-as-a-judge protocol using ChatGPT-5, Claude Sonnet 4, and Gemini 2.5 Pro.

Scoring recipe

def compute_metrics(predictions, gold, score_ids, levels):
    votes = []
    for pred, gold_ans in zip(predictions, gold):
        votes.append(1 if sum(j.is_correct(pred, gold_ans) for j in [GPT5, Claude4, Gemini25]) >= 2 else 0)
    accuracy = sum(votes) / len(votes)
    lsr = []
    for l in range(1, 5):
        correct_scores = 0
        unique_scores = set(score_ids)
        for sid in unique_scores:
            q_idx = [i for i, s in enumerate(score_ids) if s == sid and levels[i] <= l]
            if all(votes[i] for i in q_idx):
                correct_scores += 1
        lsr.append(correct_scores / len(unique_scores))
    return accuracy, lsr

Common pitfalls

• Confusing per-question accuracy with Level-wise Success Rate (LSR), which strictly requires all questions up to level l on a single score to be correct.
• Assuming exact string matching is used for scoring; the paper employs a 3-model LLM-as-a-judge voting protocol to determine correctness.
• Overlooking the significant modality gap: visual QA (PDF) performance is drastically lower than textual QA (ABC notation) due to recognition and localization errors.

Evidence (verbatim from paper)

Accuracy is reported at both the individual level and the aggregate level (overall). We consider two evaluations: (1) zero-shot, testing models directly on the 1,800 QA pairs; and (2) fine-tuned, where models are adapted with LoRA (Hu et al., 2021). We also introduce the Level-wise Success Rate (LSR), capturing the probability of correctly answering successive levels for each score.

Citation

@misc{dai2025msubench,
  title={Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores},
  author={Dai et al. (2025)},
  year={2025},
  note={arXiv:2511.20697}
}

• arXiv: 2511.20697