Muchomusic Eval

muchomusic-eval

MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models — Weck et al. (2024) (arXiv:2408.01337, 2024)

What this evaluates

Evaluates multimodal audio-language models' ability to understand music through factual knowledge and reasoning tasks. It probes whether models can ground their answers in audio content rather than relying on language priors or hallucinating musical elements.

Datasets

• MuChoMusic — total 1187; splits: test (1187)

Metrics

• accuracy (primary) — range: percent
  • Percentage of questions where the model correctly selects the ground-truth answer option from the provided multiple-choice set.
• instruction following rate (IFR) — range: percent
  • Percentage of questions where the model's output matches the expected answer format, regardless of correctness.

Input / output format

Input: A music audio clip paired with a text question and a set of multiple-choice answer options (typically 4, but can be reduced to 2 or 3 for ablation).

Output: A single selected answer option from the provided choices. Models must output the answer in a specific format to be counted as instruction-following.

Scoring recipe

def compute_metrics(predictions, golds, format_checker):
    total = len(golds)
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = (correct / total) * 100
    follows_format = sum(1 for p in predictions if format_checker(p))
    ifr = (follows_format / total) * 100
    return {'accuracy': accuracy, 'if': ifr}

Common pitfalls

• Models often fail to output answers in the correct format, which artificially lowers accuracy scores.
• Models exhibit strong language bias, answering based on text priors rather than audio content, leading to poor audio-grounded reasoning.
• Auditory and language hallucinations cause models to ignore provided options and generate uninformative or incorrect responses.

Evidence (verbatim from paper)

We report results for all models in Table 3, showing the overall accuracy score alongside detailed scores on knowledge and reasoning questions, and the instruction following rate (IFR).

Citation

@misc{weck2024muchomusic,
  title={MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models},
  author={Weck et al. (2024)},
  year={2024},
  note={arXiv:2408.01337}
}

• arXiv: 2408.01337