Music Caption Retrieval Eval

music-caption-retrieval-eval

Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation — Junda Wu et al. (arXiv:2407.20445, 2024)

What this evaluates

Evaluates a model's ability to generate fine-grained, temporally-aware music captions and retrieve corresponding audio segments using those captions. It probes the model's capacity for temporal reasoning, structural music understanding, and cross-modal alignment.

Datasets

• MusicCaps — total ?; splits: test (-1)
• Song Descriptor — total ?; splits: test (-1)

Metrics

• BLEU-1/2/3, METEOR, ROUGE-L, BERTScore (primary) — range: percent
  • Standard NLP sequence-to-sequence metrics comparing generated captions against human annotations. BLEU measures n-gram precision, METEOR adds synonymy and stemming, ROUGE-L uses longest common subsequence, and BERTScore uses contextual embeddings for semantic similarity.
• Recall@K, Median Rank (primary) — range: percent
  • For retrieval, compute IoU between caption and audio time segments. Use CLAP to extract features and calculate IoU-weighted average cosine similarity. Rank audio clips by similarity; Recall@K is the fraction of correct clips in the top K, and Median Rank is the median position of the correct clip.
• CLAP Score — range: [0, 1]
  • Measures audio-text similarity by computing the cosine similarity between audio and text embeddings extracted from the CLAP model.

Input / output format

Input: Audio clips with associated time segments and text captions (global or segment-level with time boundaries).

Output: Generated text captions (global or temporally segmented), ranked list of retrieved audio clips, or generated audio tracks.

Scoring recipe

def evaluate_caption(pred, gold):
    return bleu(pred, gold), meteor(pred, gold), rouge(pred, gold), bertscore(pred, gold)

def evaluate_retrieval(preds, gold, clips, segments):
    ious = compute_iou(preds.segments, clips.segments)
    feats = clap_encode([preds.text, clips.audio])
    sims = ious * cosine_similarity(feats.text, feats.audio)
    ranks = rank_by_similarity(sims)
    return recall_at_k(ranks, k=[1,5,10]), median_rank(ranks)

def evaluate_generation(gen_audio, prompt):
    return clap_score(gen_audio, prompt)

Common pitfalls

• Comparing FUTGA's temporally segmented captions directly to global MusicCaps captions without accounting for formal differences, which artificially penalizes the model.
• Using LLM-generated captions for retrieval without considering that CLAP's text encoder may be out-of-distribution for complex LLM language, leading to lower-than-expected retrieval scores.
• Failing to separate global vs. complete (segmented) caption evaluations, which conflates different levels of temporal granularity.

Evidence (verbatim from paper)

We follow the previous works [14, 22] and report the metrics, BLEU (B),METEOR (M), ROUGE (R), and BERT-score (B-S), in Table 3. Since our captions are formally different from original music captions, we report the evaluation metrics for the global and the complete captions in our dataset separately.

Citation

@misc{wu2024futga,
  title={Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation},
  author={Junda Wu et al.},
  year={2024},
  note={arXiv:2407.20445}
}

• arXiv: 2407.20445