Mos Bench Eval

mos-bench-eval

MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Models — Huang et al. (2024) (arXiv:2411.03715, 2024)

What this evaluates

This benchmark evaluates the out-of-domain generalization and robustness of subjective speech quality assessment (SSQA) models. It probes whether models trained on single or multiple datasets can accurately predict human-perceived quality scores across diverse conditions, including different languages, speech types (TTS, voice conversion, enhancement, noisy), and sampling frequencies.

Datasets

• MOS-Bench — total ?; splits: test (-1); repo https://github.com/unilight/sheet

Metrics

• Sys MSE — range: other
  • Mean squared error between predicted and ground truth MOS scores, averaged across systems. Used primarily for synthetic speech datasets.
• Sys SRCC — range: [-1, 1]
  • Spearman rank correlation coefficient between predicted and ground truth system-level average MOS scores. Used to evaluate ranking fidelity.
• Utt MSE — range: other
  • Mean squared error between predicted and ground truth utterance-level MOS scores. Used primarily for non-synthetic speech datasets.
• Utt LCC — range: [-1, 1]
  • Linear correlation coefficient (Pearson) between predicted and ground truth utterance-level MOS scores.
• Best score difference (primary) — range: other
  • MSE of a specific model minus the MSE of the best-performing model on the same test set. Averaged across all 12 test sets to assess overall generalization.
• Best score ratio — range: [0, 1]
  • LCC or SRCC of a specific model divided by the LCC or SRCC of the best-performing model on the same test set. Averaged across all 12 test sets.

Input / output format

Input: Raw speech audio waveform, explicitly resampled to 16 kHz.

Output: A single continuous float representing the predicted subjective mean opinion score (MOS).

Scoring recipe

def compute_metrics(preds, golds):
    mse = np.mean((preds - golds) ** 2)
    lcc = np.corrcoef(preds, golds)[0, 1]
    srcc = spearmanr(preds, golds).correlation
    return {'MSE': mse, 'LCC': lcc, 'SRCC': srcc}

def compute_best_score_diff_ratio(model_res, best_res):
    diff = model_res['MSE'] - best_res['MSE']
    ratio = model_res['LCC'] / best_res['LCC']
    return {'best_score_difference': diff, 'best_score_ratio': ratio}

Common pitfalls

• Relying solely on system-level SRCC can mask unfaithful predictions (e.g., score skewing); high MSE is required to detect this lack of faithfulness.
• Input audio must be explicitly resampled to 16 kHz, as the underlying wav2vec 2.0 encoder only accepts 16 kHz waveforms.
• The definition of the 'best performing' model varies between single-dataset and multi-dataset training experiments, which changes the baseline for best score difference/ratio calculations.

Evidence (verbatim from paper)

For datasets that contain mostly synthetic speech, following the VMC series, we mainly reported two evaluation metrics: system-level mean squared error (Sys MSE) and system-level spearman rank correlation coefficient (Sys SRCC)... Here we would like to emphasize the importance of the MSE metric. MSE is a useful metric to reflect the faithfulness... best score difference := MSE of a specific model - MSE of the best performing model, best score ratio := LCC/SRCC of a specific model / LCC/SRCC of the best performing model.

Citation

@misc{huang2024mosbench,
  title={MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Models},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2411.03715}
}

• arXiv: 2411.03715