Sidon Speech Restoration Eval

sidon-speech-restoration-eval

Sidon: Fast and Robust Open-Source Multilingual Speech Restoration for Large-scale Dataset Cleansing — Nakata et al. (2025) (arXiv:2509.17052, 2025)

What this evaluates

Evaluates multilingual speech restoration quality by measuring acoustic fidelity, speaker preservation, and transcription accuracy on noisy speech. It also assesses downstream utility by training TTS models on cleansed data and measuring synthetic speech quality, alongside inference speed benchmarks.

Datasets

• test-clean/test-other subsets (English) — total ?; splits: test-clean (-1), test-other (-1)
• Multilingual test set — total ?; splits: test (-1)
• TED-LIUM Release 3 — total ?; splits: test (-1)

Metrics

• DNSMOS (primary) — range: other
  • Deep Noise Suppression Mean Opinion Score; a learned perceptual metric predicting human-rated speech quality from the restored audio waveform.
• NISQA — range: other
  • Non-Intrusive Speech Quality Assessment; a deep learning model predicting subjective speech quality scores without requiring a reference signal.
• SpkSim — range: other
  • Speaker Similarity; cosine similarity between speaker embeddings extracted from the original and restored audio.
• WER — range: [0, 1]
  • Word Error Rate; ratio of insertions, deletions, and substitutions to the total number of words in the reference transcript.
• CER — range: [0, 1]
  • Character Error Rate; ratio of insertions, deletions, and substitutions to the total number of characters in the reference transcript.
• MOS — range: other
  • Mean Opinion Score; average subjective rating (typically 1-5) of synthetic speech quality from a TTS model trained on restored data.
• RTF — range: other
  • Real-Time Factor; ratio of processing time to audio duration (RTF < 1 indicates faster-than-real-time inference).

Input / output format

Input: Noisy speech audio (16 kHz sampling rate), typically 30-second segments for speed evaluation and variable-length for restoration evaluation.

Output: Restored speech waveform; for TTS evaluation, synthetic speech audio generated by a TTS model trained on the restored data; for speed evaluation, elapsed processing time per batch.

Scoring recipe

def evaluate_restoration(gold_audio, restored_audio, gold_transcript):
    wer = compute_wer(gold_transcript, asr_model(restored_audio))
    cer = compute_cer(gold_transcript, asr_model(restored_audio))
    dnsmos = dnsmos_predictor(restored_audio)
    nisqa = nisqa_predictor(restored_audio)
    spksim = cosine_similarity(emb(gold_audio), emb(restored_audio))
    return {'WER': wer, 'CER': cer, 'DNSMOS': dnsmos, 'NISQA': nisqa, 'SpkSim': spksim}

def evaluate_tts(cleansed_audio, tts_model):
    synth_audio = tts_model.generate(cleansed_audio)
    mos = compute_mos(synth_audio)
    return {'MOS': mos}

def evaluate_speed(elapsed_time, audio_duration):
    return {'RTF': elapsed_time / audio_duration}

Common pitfalls

• NISQA is trained exclusively on English subjective data, making its scores potentially unreliable or biased for multilingual evaluation.
• WER comparisons favor text-conditioned models (like Miipher) over text-unconditioned ones (like Sidon), as the former can leverage ground-truth transcripts.
• RTF is highly dependent on batch size and GPU architecture (measured on NVIDIA H200); results will not generalize directly to consumer GPUs or different batch configurations.

Evidence (verbatim from paper)

In terms of sound quality, a comparison of DNSMOS and NISQA scores between the noisy inputs and the Sidon outputs shows that Sidon consistently improves the sound quality in any tested language... On average, Sidon outperforms Miipher-2 in CER and DNSMOS, is comparable in SpkSim, and is slightly worse in NISQA.

Citation

@misc{nakata2025sidon,
  title={Sidon: Fast and Robust Open-Source Multilingual Speech Restoration for Large-scale Dataset Cleansing},
  author={Nakata et al. (2025)},
  year={2025},
  note={arXiv:2509.17052}
}

• arXiv: 2509.17052