Sdmuse Music Editing Eval

sdmuse-music-editing-eval

SDMuse: Stochastic Differential Music Editing and Generation via Hybrid Representation — Chen Zhang et al. (2022) (arXiv:2211.00222, 2022)

What this evaluates

Evaluates the quality and controllability of a stochastic differential music generation and editing model. It probes the model's ability to generate pop piano music from scratch or conditioned on control signals, and perform fine-grained editing tasks like stroke-based generation, inpainting, and style transfer.

Datasets

• ailabs1k7 — total 1748; splits: train (-1), test (-1)

Metrics

• pitch distribution similarity (PD) (primary) — range: [0, 1]
  • Distribution similarity metric introduced in Sheng et al. (2021) comparing the pitch distribution of generated music to a reference. Higher values indicate better quality.
• duration distribution similarity (DD) — range: [0, 1]
  • Distribution similarity metric introduced in Sheng et al. (2021) comparing the duration distribution of generated music to a reference. Higher values indicate better quality.
• control signal distance (CSD) — range: [0, inf)
  • L2 distance between the given control signal (note density or pitch distribution) and the corresponding control signal extracted from the generated output. Lower values indicate better controllability.
• overlap ratio (OR) — range: [0, 1]
  • Overlap ratio between the generated pianorolls and the input stroke pianorolls. Higher values indicate better alignment with the editing prompt.
• overall perceptive score — range: [1, 5]
  • Subjective score on a 5-point scale (1=bad, 5=excellent) provided by 10 participants with music knowledge to rate generation quality.
• overall consistency score — range: [1, 5]
  • Subjective score on a 5-point scale (1=bad, 5=excellent) provided by 10 participants with music knowledge to rate consistency in fine-grained editing tasks.

Input / output format

Input: Control signals (note density, pitch distribution, stroke pianorolls) or null for unconditioned generation. Input is a 32-bar segment of pop piano music.

Output: MIDI-event sequences or pianorolls representing the generated or edited 32-bar musical segment.

Scoring recipe

def compute_metrics(predictions, gold, control_signal):
    # PD & DD: Distribution similarity (Sheng et al. 2021)
    pd = compute_distribution_similarity(predictions.pitches, gold.pitches)
    dd = compute_distribution_similarity(predictions.durations, gold.durations)
    # CSD: L2 distance of control signals
    csd_n = np.linalg.norm(control_signal.note_density - predictions.note_density)
    csd_p = np.linalg.norm(control_signal.pitch_dist - predictions.pitch_dist)
    # OR: Overlap ratio for stroke-based tasks
    or_score = np.sum(np.logical_and(predictions.pianoroll, gold.stroke_pianoroll)) / np.sum(gold.stroke_pianoroll)
    # Subjective scores: Average of 10 annotators on 1-5 scale
    perceptive_score = np.mean(annotator_scores)
    consistency_score = np.mean(annotator_scores)
    return {'PD': pd, 'DD': dd, 'CSD_n': csd_n, 'CSD_p': csd_p, 'OR': or_score, 'Perceptive': perceptive_score, 'Consistency': consistency_score}

Common pitfalls

• PD and DD are distribution similarity metrics from Sheng et al. (2021), not standard accuracy or perplexity scores.
• CSD is computed separately for note density and pitch distribution, with lower values indicating better controllability.
• Subjective scores are averaged over 10 music-knowledge annotators using a 5-point scale, not automated metrics.

Evidence (verbatim from paper)

As listed in Table 2, to evaluate quality, we use PD and DD scores introduced in Sheng et al. (2021) and conduct subjective evaluation to obtain the overall perceptive scores. On the other hand, for quantifying controllability of conditioned music generation and editing, we calculate the L2 distance of control signals (CSD) between the given one and those of generated output. Also, when conducting fine-grained music editing like stroke-based generation, we compute the overlap ratio (OR) between the generated pianorolls and the input stroke pianorolls.

Citation

@misc{zhang2022sdmuse,
  title={SDMuse: Stochastic Differential Music Editing and Generation via Hybrid Representation},
  author={Chen Zhang et al. (2022)},
  year={2022},
  note={arXiv:2211.00222}
}

• arXiv: 2211.00222