Marble Eval

marble-eval

MARBLE: Music Audio Representation Benchmark for Universal Evaluation — Yuan et al. (2023) (arXiv:2306.10548, 2023)

What this evaluates

Evaluates pre-trained music audio representation models across a unified taxonomy of 18 downstream tasks spanning acoustic, performance, score, and high-level description levels. It assesses model generalization and representation quality under constrained training settings, including sequence labeling tasks like beat tracking and source separation.

Datasets

• MelodyDB — total ?; splits: test (-1)
• Muljam — total ?; splits: test (-1)
• Jamendo — total ?; splits: test (-1)
• GuitarSet — total ?; splits: test (-1)
• MUSDB18 — total ?; splits: test (-1)
• NSynth — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted class labels out of total instances. Standard classification accuracy.
• CER — range: [0, 1]
  • Character Error Rate: minimum number of single-character edits (insertions, deletions, substitutions) required to change the predicted sequence into the reference, normalized by reference length.
• WER — range: [0, 1]
  • Word Error Rate: minimum number of single-word edits required to change the predicted sequence into the reference, normalized by reference length.
• Chord Estimation Accuracy — range: [0, 1]
  • Task-specific accuracy for chord labels (root, majmin, mirex, thirds, sevenths, triads, inversions). Predicted chord is correct if it matches the ground truth label within a specified tolerance window.

Input / output format

Input: Audio embeddings or features extracted from pre-trained models, optionally passed through task-specific linear probes or sequence labeling heads.

Output: Task-specific predictions: discrete class labels for classification tasks, character/word sequences for transcription tasks, frame-level binary or multi-class labels for sequence labeling, and chord labels for chord estimation.

Scoring recipe

def compute_marble_metrics(preds, golds, task_type):
    if task_type == 'classification':
        return sum(p == g for p, g in zip(preds, golds)) / len(golds)
    elif task_type in ['lyrics_transcription']:
        # CER/WER: edit distance normalized by reference length
        edits = levenshtein_distance(preds, golds)
        return edits / len(golds)
    elif task_type == 'chord_estimation':
        correct = sum(1 for p, g in zip(preds, golds) if p == g)
        return correct / len(golds)
    else:
        raise ValueError('Unsupported task type')

Common pitfalls

• Excluding sequence labeling tasks (beat tracking, source separation) from overall average scores when some baseline models lack frame-level representations.
• Failing to apply the constrained training protocol, which strictly limits downstream structures and hyperparameter search spaces to ensure fair comparison.
• Calculating overall averages across all tasks without filtering for models that are actually applicable to every task.

Evidence (verbatim from paper)

For instance, the best performance on NSynth Pitch classification have achieved up to 94.4% accuracy. The overall average scores are calculated on the systems applicable to all tasks.

Citation

@misc{yuan2023marble,
  title={MARBLE: Music Audio Representation Benchmark for Universal Evaluation},
  author={Yuan et al. (2023)},
  year={2023},
  note={arXiv:2306.10548}
}

• arXiv: 2306.10548