Clamp2 Music Retrieval Eval

clamp2-music-retrieval-eval

CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models — Shangda Wu et al. (2024) (arXiv:2410.13267, 2024)

What this evaluates

Evaluates a model's ability to classify symbolic music into genres, emotions, or composer styles, and to perform cross-modal semantic search between music scores (ABC/MIDI) and textual descriptions. It also probes multilingual retrieval capabilities by testing performance across machine-translated text queries.

Datasets

• WikiMT — total 1010; splits: test (1010)
• VGMIDI — total 204; splits: test (204)
• Pianist8 — total 411; splits: test (411)
• MidiCaps — total 1010; splits: test (1010)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances in the test set.
• F1-macro — range: [0, 1]
  • Unweighted mean of recall (or F1) for each class, treating all classes equally regardless of their support.
• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: average of 1/rank for the first correctly retrieved item across all queries.
• HR@K — range: [0, 1]
  • Hit Rate at top K: fraction of queries where the relevant item appears in the top K ranked results.

Input / output format

Input: Per instance: either a symbolic music representation (ABC notation or MIDI file) for classification, or a text query paired with a candidate music piece for semantic search.

Output: Per instance: a predicted class label (genre, emotion, or composer) for classification; or a ranked list of music-text pairs for retrieval.

Scoring recipe

def compute_metrics(predictions, gold_labels, ranked_lists=None):
    if ranked_lists is None:
        correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
        accuracy = correct / len(gold_labels)
        f1_macro = average_f1_per_class(predictions, gold_labels)
        return {'accuracy': accuracy, 'f1_macro': f1_macro}
    else:
        mrr_scores = []
        hr_counts = {1: 0, 10: 0, 100: 0}
        for ranks, gold in zip(ranked_lists, gold_labels):
            rank = ranks.index(gold) + 1 if gold in ranks else len(ranks) + 1
            mrr_scores.append(1.0 / rank)
            for k in [1, 10, 100]:
                if rank <= k:
                    hr_counts[k] += 1
        mrr = sum(mrr_scores) / len(mrr_scores)
        hr = {k: v / len(gold_labels) for k, v in hr_counts.items()}
        return {'mrr': mrr, 'hr@1': hr[1], 'hr@10': hr[10], 'hr@100': hr[100]}

Common pitfalls

• Data leakage: The pre-training set includes the Lakh MIDI dataset, so the test set for MidiCaps must be carefully sampled (1,010 pieces) to avoid overlap with training data.
• Modality mismatch: Baseline models like CLaMP do not natively support MIDI, requiring conversion to ABC notation for fair comparison, which may lose performance details.
• Translation quality dependency: Multilingual retrieval results are heavily influenced by the quality of machine-translated queries, measured via BLEU scores of back-translations.

Evidence (verbatim from paper)

Table 1: Classification performance for ABC notation and MIDI was assessed across three datasets: WikiMT (1,010 pieces, 8 genres), VGMIDI (204 pieces, 4 emotions), and Pianist8 (411 pieces, 8 composers). Table 2 shows semantic search results on the WikiMT and MidiCaps benchmarks, using Mean Reciprocal Rank (MRR) and Hit Rate at Top K (HR@K) to assess model performance in retrieving and ranking relevant music-text pairs.

Citation

@misc{wu2024clamp2,
  title={CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models},
  author={Shangda Wu et al. (2024)},
  year={2024},
  note={arXiv:2410.13267}
}

• arXiv: 2410.13267