Musiccaps Eval

musiccaps-eval

MusicLM: Generating Music From Text — Andrea Agostinelli et al. (arXiv:2301.11325, 2023)

What this evaluates

Evaluates a model's ability to generate high-fidelity, long-form music from complex text descriptions. It probes both audio quality/plausibility and the model's adherence to specific textual constraints such as genre, mood, tempo, and instrumentation.

Datasets

• MusicCaps — total 5500; splits: train (-1), eval (-1), genre-balanced (1000)

Metrics

• FAD (primary) — range: other
  • Fréchet Audio Distance computed between audio embeddings (using Trill2 or VGGish models) of generated and reference clips. Lower scores indicate higher audio quality and realism.
• KLD — range: other
  • KL divergence between the class prediction probability distributions of a LEAF classifier on AudioSet for generated versus reference music. Measures adherence to acoustic characteristics implied by the text.
• MCC — range: [0, 1]
  • Average cosine similarity between MuLan embeddings of the input text captions and the generated music. Higher values indicate better text-music alignment.

Input / output format

Input: Free-text caption (average 4 sentences) describing music, including genre, mood, tempo, instrumentation, and other aspects.

Output: 10-second audio clip (generated via autoregressive token sampling with temperature 1.0/0.95/0.4 across stages).

Scoring recipe

def score(predictions, gold):
    # FAD: Fréchet distance between audio embeddings
    fad = frechet_distance(embed_audio(predictions), embed_audio(gold))
    # KLD: KL divergence of LEAF classifier predictions
    pred_probs = classifier.predict(predictions)
    gold_probs = classifier.predict(gold)
    kld = kl_divergence(pred_probs, gold_probs)
    # MCC: Cosine similarity of MuLan text/audio embeddings
    text_embs = embed_text(gold['caption'])
    music_embs = embed_audio(predictions)
    mcc = cosine_similarity(text_embs, music_embs).mean()
    return {'FAD': fad, 'KLD': kld, 'MCC': mcc}

Common pitfalls

• FAD measures audio quality/plausibility, not text adherence; a low FAD does not guarantee the generated music matches the prompt.
• KLD relies on a proxy LEAF classifier trained on AudioSet, which may not capture nuanced musical aspects or genre-specific details.
• Human evaluation explicitly instructs raters to ignore audio quality to isolate text adherence, which can skew subjective preferences.

Evidence (verbatim from paper)

We compute different metrics to evaluate MusicLM, capturing two important aspects of music generation: the audio quality and the adherence to the text description. We report the FAD based on two audio embedding models, both of which are publicly available: (1) Trill2... and (2) VGGish... we use a LEAF classifier trained for multi-label classification on AudioSet, to compute class predictions for both the generated and the reference music and measure the KL divergence between probability distributions of class predictions... define the MCC metric as the average cosine similarity between these embeddings.

Citation

@misc{agostinelli2023musiclm,
  title={MusicLM: Generating Music From Text},
  author={Andrea Agostinelli et al.},
  year={2023},
  note={arXiv:2301.11325}
}

• arXiv: 2301.11325