Wildscore Eval

wildscore-eval

WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning — Mundada et al. (2025) (arXiv:2509.04744, 2025)

What this evaluates

This benchmark evaluates multimodal large language models' ability to perform multi-step, context-sensitive reasoning over symbolic musical notation. It probes capabilities in harmonic analysis, rhythmic interpretation, structural form recognition, and expressive markings through multiple-choice questions derived from real-world compositions and forum queries.

Datasets

• WildScore — total ?; splits: test (-1); repo https://github.com/GaganVM/WildScore

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as the number of correct predictions divided by the total number of questions, multiplied by 100.

Input / output format

Input: Per instance: a symbolic score image (optional) and a multiple-choice question text. Two evaluation modalities are used: image+text and text-only.

Output: A single selected answer choice from the provided multiple-choice options.

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models may perform better in the text-only setting than in the image+text setting (e.g., InternVL, MiniCPM, LLaVA), indicating that visual context can sometimes introduce noise or alignment issues rather than aiding reasoning.
• Failures must be carefully disentangled into perception-based errors (misreading notation) versus reasoning-based errors (incorrect music-theory application), as models can correctly perceive symbols but still fail the question.
• The benchmark uses naturalized, user-generated questions where perception of the score is not always strictly necessary, meaning high accuracy might sometimes reflect textual priors rather than genuine multimodal reasoning.

Evidence (verbatim from paper)

Following standard practice in multimodal reasoning benchmarksYu et al. ([2023]), we adopt accuracy as our primary metric, calculated as the percentage of correctly answered multiple-choice questions. Each question includes one correct answer, annotated based on human or language model preference as detailed in Section [3].

Citation

@misc{mundada2025wildscore,
  title={WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning},
  author={Mundada et al. (2025)},
  year={2025},
  note={arXiv:2509.04744}
}

• arXiv: 2509.04744