Globaldisco Eval

globaldisco-eval

Bias beyond Borders: Global Inequalities in AI-Generated Music — Solak et al. (2025) (arXiv:2510.01963, 2025)

What this evaluates

Evaluates AI music generation models for global and cultural bias by measuring how well generated tracks match reference tracks across different world regions and genres. It probes the models' out-of-distribution capabilities and tendency to default to mainstream styles over authentic regional ones.

Datasets

• GlobalDISCO — total 166000; splits: test (-1)

Metrics

• FAD (primary) — range: other
  • Fréchet Audio Distance: computes the Fréchet distance between the multivariate Gaussian distributions of audio features extracted by embedding models (PANNs, CLAP, MUQ-MULAN) for generated and reference tracks. Lower scores indicate better fidelity.
• KAD — range: other
  • Kernel Audio Distance: measures the distributional distance between generated and reference audio tracks using kernel methods. Lower scores indicate better alignment with reference distributions.

Input / output format

Input: Audio generation prompts or genre/region specifications.

Output: Generated audio tracks.

Scoring recipe

features_gen = extract_features(generated_tracks, embedding_model)
features_ref = extract_features(reference_tracks, embedding_model)
fad = frechet_distance(features_gen, features_ref)
kad = kernel_audio_distance(features_gen, features_ref)
# Normalize scores per embedding model across genres, then average across all embedding and generation models

Common pitfalls

• FAD and KAD scores are normalized (z-scored) per embedding model before averaging, not computed globally across all tracks.
• Scores are averaged across multiple embedding models (PANNs, CLAP, MUQ-MULAN) and multiple generation models, which can mask per-model performance.
• Regional genres are selected via a tf-idf-like method based on artist frequency, not simply by country name.

Evidence (verbatim from paper)

We first explore the difference in music generation quality across different world sub-regions, as defined by the UN M49 Standard*[[26]]*. For the PANNs, CLAP, and MUQ-MULAN embedding models, we present FAD scores between generated and reference tracks for the 13 world regions present in GlobalDISCO, shown as a heatmap in [Fig.˜4]. We show the same analysis with KAD in [Fig.˜5].

Citation

@misc{solak2025biasbeyondborders,
  title={Bias beyond Borders: Global Inequalities in AI-Generated Music},
  author={Solak et al. (2025)},
  year={2025},
  note={arXiv:2510.01963}
}

• arXiv: 2510.01963