Tinymu Music Eval

tinymu-music-eval

TinyMU: A Compact Audio-Language Model for Music Understanding — Li et al. (2026) (arXiv:2604.15849, 2026)

What this evaluates

Evaluates a compact audio-language model's ability to perform music information retrieval (genre and instrument classification), generate descriptive music captions, and answer complex multiple-choice questions about musical theory and structure.

Datasets

• GTZAN — total 1000; splits: test (1000)
• Medley-Solos-DB — total 12236; splits: test (12236)
• MusicCaps — total 2800; splits: test (2800)
• MuChoMusic — total 1187; splits: test (1187)

Metrics

• classification accuracy (primary) — range: percent
  • Percentage of correctly predicted labels. For MIR tasks, the model's text output is matched to candidate labels using CLAP text encoder similarity; the highest-scoring label is selected as the prediction.
• METEOR — range: [0, 1]
  • Automatic evaluation metric for text generation that measures lexical overlap, synonymy, and stemming between generated and reference captions.
• BERTScore — range: [0, 1]
  • Evaluates text generation by computing cosine similarity between contextual embeddings of generated and reference texts using a BERT model.
• accuracy — range: percent
  • Percentage of correctly answered multiple-choice questions on the reasoning benchmark.

Input / output format

Input: Audio clip paired with a text prompt (candidate labels for classification, open-ended question for captioning, or multiple-choice question for reasoning).

Output: Text string corresponding to the predicted label, generated caption, or selected answer choice.

Scoring recipe

# MIR Tasks (GTZAN, Medley-Solos-DB)
pred = max(candidates, key=lambda c: clap_text_encoder_similarity(model_output, c))
acc = 1.0 if pred == gold_label else 0.0

# Captioning (MusicCaps)
meteor = compute_meteor(generated_caption, gold_caption)
bertscore = compute_bertscore(generated_caption, gold_caption)

# Reasoning (MuChoMusic)
acc = 1.0 if predicted_choice == gold_choice else 0.0

Common pitfalls

• MIR tasks rely on CLAP text encoder similarity to match model outputs to candidate labels rather than exact string matching, which can cause mismatches if the model paraphrases the label.
• The evaluation assumes zero-shot transfer for GTZAN and Medley-Solos-DB as they do not overlap with training data, but dataset splits are not explicitly detailed beyond 'test'.
• Freezing the language model severely degrades QA performance; full fine-tuning or LoRA is required for reasoning tasks, as shown in ablation studies.

Evidence (verbatim from paper)

Following prior work [[18], [20]], we compute the similarity between TinyMU’s outputs and candidate labels using the text encoder of CLAP [[14]], and select the highest-scoring label as model output. We then report classification accuracy for these two benchmarks. We report the commonly adopted metrics: METEOR [[4]], and BERTScore [[31]]. We report the accuracy of models on the benchmark.

Citation

@misc{li2026tinymu,
  title={TinyMU: A Compact Audio-Language Model for Music Understanding},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.15849}
}

• arXiv: 2604.15849