Cmi Rewardbench Eval

cmi-rewardbench-eval

CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction — Ma et al. (2026) (arXiv:2603.00610, 2026)

What this evaluates

Evaluates music reward models on their ability to align with human aesthetic judgments and follow compositional multimodal instructions (text, lyrics, audio). It probes both absolute musicality scoring and relative pairwise preference ranking across diverse generation models.

Datasets

• PAM — total 500; splits: test (500); repo https://github.com/Haiwen-Xia/CMI-RewardBench
• MusicEval — total 413; splits: test (413); repo https://github.com/Haiwen-Xia/CMI-RewardBench
• Music Arena — total 1340; splits: test (1340); repo https://github.com/Haiwen-Xia/CMI-RewardBench
• CMI-Pref — total 500; splits: test (500); repo https://github.com/Haiwen-Xia/CMI-RewardBench

Metrics

• Linear Correlation Coefficient (LCC) (primary) — range: [-1, 1]
  • Pearson correlation between predicted scores and human MOS/preferences. Measures linear trend alignment.
• Spearman Rank Correlation (SRCC) (primary) — range: [-1, 1]
  • Rank-based correlation between predicted scores and human MOS/preferences. Measures monotonic trend alignment.
• Kendall-Tau (K-Tau) (primary) — range: [-1, 1]
  • Rank correlation measuring the number of concordant vs discordant pairs between predictions and human labels.
• Pairwise Accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted preferred audio samples in pairwise comparisons against expert annotations.

Input / output format

Input: Compositional prompt consisting of optional text description (t), optional lyrics (l), and optional reference audio (a_ref), paired with the target evaluation audio (a_eval).

Output: Two scalar scores (s_MUS, s_ALI) ∈ ℝ² representing predicted musicality and alignment scores. For pairwise evaluation, the model outputs a preference decision indicating which of two candidate audios is superior.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions: list of (s_MUS, s_ALI) or pairwise choices
    # gold: list of human MOS or preferred audio index
    scores = [p[0] for p in predictions]  # use MUS score or average
    # Regression metrics (PAM, MusicEval)
    lcc = pearsonr(scores, gold)
    srcc = spearmanr(scores, gold)
    ktau = kendalltau(scores, gold)
    # Pairwise accuracy (Music Arena, CMI-Pref)
    correct = sum(1 for pred, gold_pair in zip(predictions, gold) if pred == gold_pair)
    accuracy = correct / len(gold)
    return {"LCC": lcc, "SRCC": srcc, "K-Tau": ktau, "Accuracy": accuracy}

Common pitfalls

• Using MSE instead of correlation metrics, as score ranges vary significantly across different generation models and datasets.
• Failing to separate musicality (MUS) and alignment (ALI) dimensions, which are evaluated independently.
• Including tied preferences in pairwise accuracy calculations, which the protocol explicitly excludes.

Evidence (verbatim from paper)

For the PAM and MusicEval datasets, we evaluate reward models on absolute musicality and alignment scoring. Given that different models and datasets utilize varying score ranges, Mean Squared Error (MSE) is insufficient for measuring generalization. Instead, we prioritize relative trend alignment using Linear Correlation Coefficient (LCC) , Spearman Rank Correlation (SRCC) , and Kendall-Tau (K-Tau). For Music Arena and the CMI-Pref test split, we evaluate models on pairwise preference accuracy. Models must determine which of two audio samples is superior in terms of musicality or better aligned with the provided compositional instructions. Accuracy is calculated by comparing model predictions against experts’ annotation.

Citation

@misc{ma2026cmirewardbench,
  title={CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction},
  author={Ma et al. (2026)},
  year={2026},
  note={arXiv:2603.00610}
}

• arXiv: 2603.00610