Mtat Nsynth Fma Eval

mtat-nsynth-fma-eval

Multi-Source Contrastive Learning from Musical Audio — Garoufis et al. (2023) (arXiv:2302.07077, 2023)

What this evaluates

Evaluates the quality of self-supervised audio representations on three downstream tasks: music auto-tagging, instrument family classification, and music genre classification. It probes how well contrastive learning objectives capture semantic and structural musical features.

Datasets

• MTAT — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/jongpillee/music_dataset_split
• NSynth — total 305979; splits: train (289205), val (12678), test (4096)
• FMA — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Macro average Receiver Operating Characteristic Area Under the Curve computed over all tags for multi-label auto-tagging.
• PR-AUC — range: [0, 1]
  • Macro average Precision-Recall Area Under the Curve computed over all tags for multi-label auto-tagging.
• WA (%) — range: percent
  • Weighted Accuracy percentage computed over classes for instrument family and genre classification.

Input / output format

Input: 1-second audio segments extracted from tracks.

Output: Per-segment prediction probabilities aggregated via averaging to produce instance-level predictions for multi-label tags or class labels.

Scoring recipe

def evaluate(segments, y_true, is_multilabel):
    # Aggregate per-segment predictions to instance level
    preds = average_segment_predictions(segments)
    
    if is_multilabel:
        roc_auc = macro_roc_auc(y_true, preds)
        pr_auc = macro_pr_auc(y_true, preds)
        return roc_auc, pr_auc
    else:
        wa = weighted_accuracy(y_true, preds)
        return wa

# Report mean over 5 independent runs with different random seeds

Common pitfalls

• Models receive 1-sec segments, but evaluation aggregates predictions per instance (track) by averaging probabilities, not by majority vote or argmax.
• Metrics are macro-averaged over all tags/classes, not micro-averaged or per-class.
• Results are averaged over 5 independent runs with different random seeds, not a single split.

Evidence (verbatim from paper)

Since music auto-tagging constitutes a multi-instance multi-label task, we use the categorical cross-entropy as its loss function, and report on the macro average ROC-AUC and PR-AUC values over all tags. On the other hand, for both instrument family classification and music genre classification, the binary cross-entropy is used as the loss function and the weighted accuracy (WA, %) as the evaluation metric. During evaluation, all models receive 1-sec audio segments, and aggregate the predictions for each instance via averaging the per-segment predictions. We further note that all downstream classification experiments were repeated 5 times, and we report on their average scores.

Citation

@misc{garoufis2023multisourcecontrastive,
  title={Multi-Source Contrastive Learning from Musical Audio},
  author={Garoufis et al. (2023)},
  year={2023},
  note={arXiv:2302.07077}
}

• arXiv: 2302.07077