Musicsem Eval

musicsem-eval

MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions — Salganik et al. (2026) (arXiv:2602.17769, 2026)

What this evaluates

Evaluates multimodal models on their ability to understand, generate, and retrieve music based on semantically rich, context-aware natural language descriptions. It probes fine-grained musical semantics beyond technical attributes, including atmospheric, situational, and contextual cues.

Datasets

• MusicSem — total ?; splits: test (-1), train (-1); repo https://github.com/Rsalganik1123/MusicSem

Metrics

• BLEU (primary) — range: [0, 1]
  • Measures n-gram precision (typically unigrams, bigrams, and trigrams) between the generated caption and the ground-truth description.
• ROUGE — range: [0, 1]
  • Measures recall-based overlap of n-grams between the generated and reference captions.
• METEOR — range: [0, 1]
  • A metric that combines precision and recall with synonymy and stemming matching to evaluate semantic similarity.

Input / output format

Input: Text-to-Music: text prompt. Music-to-Text: audio clip. Retrieval: paired text and audio.

Output: Text-to-Music: audio waveform. Music-to-Text: natural language caption. Retrieval: similarity score or ranked list.

Scoring recipe

def compute_metrics(predictions, references):
    bleu = nltk.bleu_score.corpus_bleu(references, predictions)
    rouge = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)
    rouge_scores = [rouge.score(ref, pred)['rouge1'].fmeasure for ref, pred in zip(references, predictions)]
    meteor = compute_meteor(predictions, references)
    return {'bleu': bleu, 'rouge': sum(rouge_scores)/len(rouge_scores), 'meteor': meteor}

Common pitfalls

• MusicCaps and Song Describer lack official train/test splits, so models are evaluated on the full public data, risking overfitting assessment.
• Generation duration varies significantly across models, making direct latency comparisons difficult without normalizing by output length.
• Proprietary models (e.g., Mureka) are evaluated via API, limiting reproducibility of exact generation parameters.

Evidence (verbatim from paper)

For all evaluations on MusicSem, we use only this held-out portion of the data for testing, while releasing the remaining entries as the public training set. ... Following canonical works in music-to-text generation, we first consider three n-gram-based metrics originally developed for machine translation: BLEU, ROUGE, and METEOR. BLEU (B) measures precision by computing the overlap of n-grams (typically unigrams, bigrams, and trigrams; i.e., B1, B2, and B3) between the

Citation

@misc{salganik2026musicsem,
  title={MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions},
  author={Salganik et al. (2026)},
  year={2026},
  note={arXiv:2602.17769}
}

• arXiv: 2602.17769