Cmi Bench Eval

cmi-bench-eval

CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following — Ma et al. (2025) (arXiv:2506.12285, 2025)

What this evaluates

Evaluates audio-text LLMs on music instruction following by framing traditional music information retrieval (MIR) tasks as prompts. It measures how accurately models follow instructions to perform classification, regression, captioning, and sequential audio analysis tasks.

Datasets

• CMI-Bench — total ?; splits: test (-1); repo https://github.com/nicolaus625/CMI-bench

Metrics

• Accuracy (primary) — range: [0, 1]
  • Strict string matching: prediction is correct if it contains only the correct label (case-, space-, and punctuation-insensitive) and no others.
• ROC/PR — range: [0, 1]
  • Cosine similarity between BGE-embedded predicted and ground-truth tag sets, used to compute Receiver Operating Characteristic and Precision-Recall curves.
• Weighted Score — range: [0, 1]
  • Standard metric from mir_eval.key that accounts for musically reasonable errors like relative minor or parallel key.
• R² — range: [-1, 1]
  • Coefficient of determination computed between z-score normalized predictions and annotations for arousal and valence.
• BLEU/METEOR/ROUGE/Bert-Score — range: [0, 1]
  • Standard NLP metrics for assessing music caption quality against ground-truth text.
• WER/CER — range: [0, 1]
  • Word Error Rate and Character Error Rate computed against ground-truth lyrics after removing typical prefixes.
• F-measure — range: [0, 1]
  • Beat tracking F-measure from mir_eval.beat with a 20ms tolerance window.
• Frame-level Accuracy — range: [0, 1]
  • Melody extraction accuracy using mir_eval.melody.evaluate with 50 music cents tolerance.
• Macro/Micro-F1 — range: [0, 1]
  • Frame-level F1 scores for instrument playing technique detection, allowing overlapping techniques.

Input / output format

Input: Audio clip paired with a natural language instruction specifying the MIR task (e.g., genre classification, key detection, lyrics transcription, or melody extraction).

Output: Text response conforming to task-specific formats: single label, list of tags, integer, descriptive caption, list of time points, or list of (time, pitch) tuples.

Scoring recipe

def evaluate(pred, gold, task):
    p = postprocess(pred)  # filter invalids, remove prefixes, floor floats, sort times
    if task == 'cls': return 1.0 if strict_match(p, gold) else 0.0
    if task == 'ml_cls': return compute_roc_pr(cosine_sim(embed(p), embed(gold)))
    if task == 'key': return mir_eval.key.weighted_score(p, gold)
    if task == 'reg': return r2_score(zscore(p), gold)
    if task == 'caption': return [bleu(p,gold), meteor(p,gold), rouge(p,gold), bertscore(p,gold)]
    if task == 'lyrics': return wer_cer(p, gold)
    if task == 'beat': return mir_eval.beat.f_measure(p, gold, tolerance=0.02)
    if task == 'melody': return mir_eval.melody.evaluate(p, gold, tolerance=50)
    if task == 'technique': return macro_micro_f1(p, gold)

Common pitfalls

• Models often return free-form text, synonyms, or invalid formats (e.g., floats instead of integers, malformed tuples) that must be strictly filtered or converted before scoring.
• Evaluation tolerances (20ms for beats, 50 cents for melody) are critical; ignoring them or using default MIR library settings will yield incorrect scores.
• For multi-label tasks, strict string matching fails; the protocol requires embedding-based semantic matching using the BGE encoder.

Evidence (verbatim from paper)

We evaluate using strict string matching: a model’s response is considered correct if it contains only the correct label (case-, space-, and punctuation-insensitive) and no others. For pitch classification, we additionally require the model to follow the instruction format and return MIDI numbers. Accuracy is used as the metric.

Citation

@misc{ma2025cmibench,
  title={CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following},
  author={Ma et al. (2025)},
  year={2025},
  note={arXiv:2506.12285}
}

• arXiv: 2506.12285