Music Controlnet Eval

music-controlnet-eval

Music ControlNet: Multiple Time-varying Controls for Music Generation — Shih-Lun Wu et al. (2023) (arXiv:2311.07069, 2023)

What this evaluates

This evaluation probes a diffusion-based music generation model's ability to precisely follow time-varying control signals (melody, dynamics, rhythm) and global style tags (genre/mood). It measures how faithfully the generated audio adheres to these inputs while maintaining overall audio realism and diversity.

Datasets

• In-domain test set — total 2000; splits: test (2000)
• MusicCaps — total 5000; splits: test (5000)
• MusicCaps+ChatGPT — total 5000; splits: test (5000)
• Created Controls dataset — total 6400; splits: test (6400)

Metrics

• Melody accuracy (primary) — range: percent
  • Percentage of frame-wise pitch classes (out of 12) that match between the input melody control and the pitch classes extracted from the generated audio.
• Dynamics correlation — range: [-1, 1]
  • Pearson’s correlation coefficient between input dynamics values and those computed from the generation. Computed per-generation (micro) or pooled across all generations (macro).
• Rhythm F1 — range: percent
  • F1 score measuring alignment between beat/downbeat timestamps from the input control and the generation. Timestamps are considered aligned if they differ by less than 70 milliseconds.
• CLAP score — range: [-1, 1]
  • Cosine similarity between text embeddings (formatted as 'An audio of [mood] [genre] music') and audio embeddings extracted from the generated audio using the CLAP dual-encoder model.
• FAD — range: other
  • Fréchet Audio Distance measuring the distributional distance between embeddings of reference audios (typically the in-domain test set) and generated audios using a VGGish model. Lower is better.

Input / output format

Input: Mel-scaled spectrograms (160 bins, ~86 Hz frame rate), global style embeddings (genre/mood), and time-varying control signals (melody pitch classes, dynamics curves, or rhythm/downbeat probabilities).

Output: Generated audio waveform (22.05 kHz) produced by a DiffWave vocoder from the model's predicted spectrogram.

Scoring recipe

def score_melody_acc(input_pitch, gen_pitch):
    return sum(1 for i in range(len(input_pitch)) if input_pitch[i] == gen_pitch[i]) / len(input_pitch)

def score_dynamics_corr(input_dyn, gen_dyn):
    return pearsonr(input_dyn, gen_dyn).statistic

def score_rhythm_f1(input_beats, gen_beats, threshold=0.07):
    tp = sum(1 for g in gen_beats if any(abs(g - i) < threshold for i in input_beats))
    prec = tp / len(gen_beats) if gen_beats else 0
    rec = tp / len(input_beats) if input_beats else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def score_clap(text_prompt, gen_audio):
    return cosine_similarity(clap_text_embed(text_prompt), clap_audio_embed(gen_audio))

def score_fad(ref_audios, gen_audios):
    return frechet_distance(vggish_embed(ref_audios), vggish_embed(gen_audios))

Common pitfalls

• Micro vs. macro dynamics correlation are computed differently (per-sample vs. pooled across all samples); mixing them invalidates cross-paper comparisons.
• FAD scores are highly sensitive to the reference dataset; the paper specifies using the in-domain test set, so using a different reference will produce incomparable scores.
• Rhythm F1 relies on a strict 70ms alignment threshold; relaxing this threshold significantly inflates scores and deviates from the paper's protocol.
• CLAP evaluation requires the exact prompt template 'An audio of [mood] [genre] music'; using raw free-form text will yield lower, non-comparable adherence scores.

Evidence (verbatim from paper)

We use the following metrics to evaluate time-varying controllability, adherence to global text (i.e., mood & genre tags) control, and overall audio realism. Melody accuracy examines whether the frame-wise pitch classes (C, C#,…, B; 12 in total) match between the input melody control and that extracted from the generation. Dynamics correlation is the Pearson’s correlation between the frame-wise input dynamics values to the values computed from the generation. Rhythm F1 follows the standard evaluation methodology for beat/downbeat detection. It quantifies the alignment between the beat/downbeat timestamps estimated from the input rhythm control, and those from the generation.

Citation

@misc{wu2023musiccontrolnet,
  title={Music ControlNet: Multiple Time-varying Controls for Music Generation},
  author={Shih-Lun Wu et al. (2023)},
  year={2023},
  note={arXiv:2311.07069}
}

• arXiv: 2311.07069