Hummusqa Eval

hummusqa-eval

HumMusQA: A Human-written Music Understanding QA Benchmark Dataset — Weck et al. (2026) (arXiv:2603.27877, 2026)

What this evaluates

Evaluates large audio-language models on music understanding by testing their ability to answer multiple-choice questions about audio excerpts. It probes perceptual grounding, structural/harmonic/cultural reasoning, and robustness against text-only shortcuts or answer-position bias.

Datasets

• HumMusQA — total 320; splits: test (320)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly matched answers after parsing model outputs with an LLM against the ground truth options. Computed by averaging results across four independent evaluation runs per question, where answer options are randomly shuffled in each run.

Input / output format

Input: Audio snippet paired with a multiple-choice question containing four answer options.

Output: Free-text response from the model, which is automatically parsed and matched to one of the four options by an LLM (gemini-2.5-flash).

Scoring recipe

def compute_accuracy(raw_outputs, gold_options, gold_answer):
    # raw_outputs: list of 4 model responses (one per shuffled-option run)
    # gold_options: list of 4 option strings
    # gold_answer: correct option string
    matched = []
    for output in raw_outputs:
        # LLM (gemini-2.5-flash) parses output to match closest option
        parsed = llm_match(output, gold_options)
        matched.append(parsed == gold_answer)
    return sum(matched) / len(matched)

Common pitfalls

• Failing to average over multiple randomized answer orderings leads to unstable performance metrics due to strong position bias in LALMs.
• Assuming high accuracy implies genuine audio perception, as models can exploit text-only cues, distractor plausibility, and statistical likelihood to answer correctly without audio.

Evidence (verbatim from paper)

Models are evaluated by prompting them with audio snippets and corresponding multiple-choice questions. Previous studies have shown that both large audio-language models and text-only large language models are highly sensitive to the ordering of multiple-choice options, with answer position alone inducing substantial performance variance and unstable model rankings. To address this issue, and following established practices in recent audio and music understanding benchmarks, we evaluate each model under multiple randomized answer orderings. Specifically, for each question, we perform four independent evaluation runs, where the answer options are randomly shuffled in each run. Final performance metrics are computed by averaging results across these runs. The output provided by the model is automatically parsed by an LLM (gemini-2.5-flash) prompted to match the response with the given options. From this matching, we calculate a simple accuracy scores which are presented in Table 1.

Citation

@misc{weck2026hummusqa,
  title={HumMusQA: A Human-written Music Understanding QA Benchmark Dataset},
  author={Weck et al. (2026)},
  year={2026},
  note={arXiv:2603.27877}
}

• arXiv: 2603.27877