Musicgen Eval

musicgen-eval

Simple and Controllable Music Generation — Copet et al. (2023) (arXiv:2306.05284, 2023)

What this evaluates

Evaluates the capability of text-to-music generation models to produce high-fidelity, controllable audio that aligns with textual descriptions and matches human perceptual quality standards.

Datasets

• MusicCaps — total 5500; splits: test (5500), genre-balanced (1000)

Metrics

• FAD (primary) — range: other
  • Computes the Fréchet distance between feature distributions of reference and generated audio using a VGGish model. Lower scores indicate more plausible audio.
• KL — range: other
  • Measures Kullback-Leibler divergence between label probability distributions predicted by an AudioSet classifier for reference and generated audio. Lower scores indicate better concept alignment.
• CLAP — range: [0, 1]
  • Cosine similarity between text and audio embeddings from a pretrained CLAP model. Higher scores indicate better audio-text alignment.
• Ovl — range: other
  • Mean human rating (1-100) for overall perceptual quality after CrowdMOS outlier filtering.
• Rel — range: other
  • Mean human rating (1-100) for text relevance after CrowdMOS outlier filtering.

Input / output format

Input: Text description (optionally merged with musical metadata tags) and/or chromagram for melody conditioning.

Output: 10-second audio waveform sampled at 32 kHz.

Scoring recipe

# Extract 10s clips from gold and predicted audio
ref_clips = extract_clips(gold_audio, duration=10)
gen_clips = extract_clips(pred_audio, duration=10)

# FAD (lower is better)
ref_feats = vggish_model(ref_clips)
gen_feats = vggish_model(gen_clips)
fad = frechet_distance(ref_feats, gen_feats)

# KL Divergence (lower is better)
ref_probs = audioset_classifier(ref_clips)
gen_probs = audioset_classifier(gen_clips)
kl = kl_divergence(ref_probs, gen_probs)

# CLAP Score (higher is better)
text_emb = clap_text_encoder(text_prompt)
audio_emb = clap_audio_encoder(pred_audio)
clap_score = cosine_similarity(text_emb, audio_emb)

# Human Ratings (Ovl, Rel)
raw_ratings = collect_mturk_ratings(pred_audio, scale=100)
clean_ratings = crowdmos_filter(raw_ratings)
ovl = mean(clean_ratings['overall_quality'])
rel = mean(clean_ratings['text_relevance'])

Common pitfalls

• FAD and KL are lower-is-better metrics, while CLAP and human scores are higher-is-better; inverting the direction leads to incorrect model ranking.
• Human evaluations require loudness normalization to -14 dB LUFS and strict CrowdMOS filtering (removing annotators rating references <85 or not listening fully) to ensure valid scores.
• The 1K genre-balanced subset is reserved for qualitative/human studies, while the full 5.5K set is used for objective metrics; mixing them invalidates the reported baselines.

Evidence (verbatim from paper)

For the objective methods, we use three metrics: the Fréchet Audio Distance (FAD), the Kullback-Leiber Divergence (KL) and the CLAP score (CLAP). We report the FAD using the official implementation in Tensorflow with the VGGish model. A low FAD score indicates the generated audio is plausible. Following Kreuk et al. (2022), we use a state-of-the-art audio classifier trained for classification on AudioSet to compute the KL-divergence over the probabilities of the labels between the original and the generated music. The generated music is expected to share similar concepts with the reference music when the KL is low. Last, the CLAP score is computed between the track description and the generated audio to quantify audio-text alignment, using the official pretrained CLAP model. For the human studies, we follow the same setup as in Kreuk et al. (2022). We ask human raters to evaluate two aspects of the audio samples (i) overall quality (Ovl), and (ii) relevance to the text input (Rel). For the overall quality test, raters were asked to rate the perceptual quality of the provided samples in a range of 1 to 100. For the text relevance test, raters were asked to rate the match between aud

Citation

@misc{copet2023simple,
  title={Simple and Controllable Music Generation},
  author={Copet et al. (2023)},
  year={2023},
  note={arXiv:2306.05284}
}

• arXiv: 2306.05284