Musicscore Eval

musicscore-eval

MusicScore: A Dataset for Music Score Modeling and Generation — Lin et al. (2024) (arXiv:2406.11462, 2024)

What this evaluates

Evaluates the ability of text-to-image generative models to produce visually coherent and structurally plausible music score images conditioned on textual descriptions of musical attributes like instrumentation, key, and composer. It benchmarks visual fidelity and distribution matching against ground-truth sheet music.

Datasets

• MusicScore-400 — total 400; splits: test (400); HF ZheqiDAI/MusicScore
• MusicScore-14k — total 14000; splits: test (14000); HF ZheqiDAI/MusicScore
• MusicScore-200k — total 200000; splits: test (200000); HF ZheqiDAI/MusicScore

Metrics

• FID (primary) — range: other
  • Fréchet Inception Distance measures the distance between two multivariate Gaussians fitted to features extracted by a pre-trained Inception network from real and generated images. Lower scores indicate better visual fidelity and distribution matching.

Input / output format

Input: Text prompt specifying musical attributes (e.g., instrumentation, key, composer, style, genre).

Output: 512×512 RGB image of a music score.

Scoring recipe

def compute_fid(predictions, gold):
    # Resize ground truth to match generation resolution
    gold_resized = [resize(img, (512, 512)) for img in gold]
    # Extract Inception features
    pred_feats = inception_model(predictions)
    gold_feats = inception_model(gold_resized)
    # Compute means and covariances
    mu_p, sigma_p = np.mean(pred_feats, axis=0), np.cov(pred_feats, rowvar=False)
    mu_g, sigma_g = np.mean(gold_feats, axis=0), np.cov(gold_feats, rowvar=False)
    # Compute FID
    diff = mu_p - mu_g
    covmean = scipy.linalg.sqrtm(sigma_p.dot(sigma_g))
    fid = np.sum(diff**2) + np.trace(sigma_p + sigma_g - 2*covmean)
    return np.real(fid)

Common pitfalls

• FID measures visual distribution similarity, not musical correctness or playability.
• Scores are highly sensitive to the number of evaluated images (FID-8 vs FID-64), so small sample sizes can yield unstable metrics.
• Resizing ground-truth scores to 512×512 may blur fine musical notation details, artificially affecting FID calculation.

Evidence (verbatim from paper)

We measure the performance of music score generation using Fréchet Inception Distance (FID) [65], which is a standard metric for evaluating generative models of images. In Table 1, FID-n indicates we randomly select n images for each evaluation, where n ∈ [8, 16, 32, 64]. The ground-truth images from each subsets are resized to same resolution with generated images at 512 × 512. During generation, we apply a DDIM sampler [67] for 250 DDIM sampling steps. The text prompt formats remain consistent between training and generation phases. We apply classifier-free guidance [68] with guidance strength ω = 4.0 in generation.

Citation

@misc{lin2024musicscore,
  title={MusicScore: A Dataset for Music Score Modeling and Generation},
  author={Lin et al. (2024)},
  year={2024},
  note={arXiv:2406.11462}
}

• arXiv: 2406.11462