Fma Genre Classification Eval

fma-genre-classification-eval

FMA: A Dataset For Music Analysis — Defferrard et al. (2016) (arXiv:1612.01840, 2016)

What this evaluates

Evaluates music information retrieval models on genre classification tasks using a large-scale, open music dataset. It probes the model's ability to map audio tracks to hierarchical genre labels (single-label or multi-label) using raw audio or precomputed features.

Datasets

• FMA (Free Music Archive) — total 106574; splits: train (-1), val (-1), test (-1); repo https://github.com/mdeff/fma

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted genre labels out of the total number of test instances.

Input / output format

Input: 30-second MP3 audio clips (for Large/Medium/Small subsets) or full-length MP3s (for Full subset), optionally accompanied by precomputed librosa features (518 features) and metadata.

Output: A predicted genre label (or set of labels) from the 161-category hierarchical taxonomy.

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Artist leakage: using tracks from the same artist in both train and test sets inflates accuracy.
• Genre hierarchy ambiguity: tracks can have multiple sub-genres; models must decide whether to predict top-level, specific, or all ancestor genres.
• Metadata noise: artist-provided genres may be inconsistent or motivated by play counts rather than objective classification.

Evidence (verbatim from paper)

It has been shown that the use of songs from the same artist in both training and test sets leads to over-optimistic accuracy and may favor some approaches

Citation

@misc{defferrard2016fma,
  title={FMA: A Dataset For Music Analysis},
  author={Defferrard et al. (2016)},
  year={2016},
  note={arXiv:1612.01840}
}

• arXiv: 1612.01840