qhjqhj00 mos-rmbench-eval
From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling — Cao et al. (2025) (arXiv:2510.00743, 2025)
What this evaluates
Evaluates the ability of speech quality reward models to correctly rank pairs of audio samples based on their Mean Opinion Score (MOS). It probes fine-grained perceptual discrimination and cross-dataset generalization in preference-based audio modeling.
Datasets
- BVCC — total ?; splits: test (-1)
- NISQA — total ?; splits: test (-1)
- SingMOS — total ?; splits: test (-1)
- SOMOS — total ?; splits: test (-1)
- TMHINT-QI — total ?; splits: test (-1)
- VMC’23 — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly ranked audio pairs out of the total evaluated pairs. Calculated as (correct_predictions / total_pairs) * 100.
Input / output format
Input: Pairs of audio samples along with their associated MOS scores (or MOS difference information for reward shaping).
Output: A binary preference prediction indicating which of the two audio samples has the higher MOS, or a continuous score used to derive the ranking.
Scoring recipe
correct = 0
total = 0
for pair in dataset:
pred = model.predict(pair.audio1, pair.audio2)
gold = pair.audio1 if pair.mos1 > pair.mos2 else pair.audio2
if pred == gold:
correct += 1
total += 1
accuracy = (correct / total) * 100
Common pitfalls
- Models struggle significantly with fine-grained discrimination when the MOS difference between paired samples is small (<0.5), leading to error rates exceeding 40%.
- MOS prediction models (e.g., UTMOS) often fail to generalize across different datasets, showing high accuracy on their training domain but dropping sharply on out-of-domain benchmarks.
- Evaluating on absolute MOS scores rather than relative preferences can misrepresent model capability, as reward modeling focuses on ranking order.
Evidence (verbatim from paper)
As shown in the evaluation results, the Classic scalar models achieve the highest overall accuracy (80.04% with BT loss), followed by the Cloud semi-scalar models (78.82% with BT loss), while the GRMs attain slightly lower overall performance.
Citation
@misc{cao2025mosrmbench,
title={From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling},
author={Cao et al. (2025)},
year={2025},
note={arXiv:2510.00743}
}
- arXiv: 2510.00743