Rats Noisy Speech Eval

rats-noisy-speech-eval

DENT-DDSP: Data-efficient noisy speech generator using differentiable digital signal processors for explicit distortion modelling and noise-robust speech recognition — Guo et al. (2022) (arXiv:2208.00987, 2022)

What this evaluates

Evaluates the fidelity of a simulated noisy speech generator against real VHF/UHF transmitted audio, and measures the downstream robustness of automatic speech recognition (ASR) models trained on the simulated data.

Datasets

• RATS Channel A — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MSSL (primary) — range: other
  • Multi-Scale Spectral Loss comparing real and simulated noisy audio. Lower values indicate higher simulation fidelity.
• WER (primary) — range: percent
  • Word Error Rate calculated on ASR predictions versus ground truth transcripts. Lower values indicate better recognition accuracy.

Input / output format

Input: Clean speech audio (1-second chunks) for distortion modeling; clean speech audio and transcribed text for ASR training/testing.

Output: Simulated noisy audio waveform; ASR predictions (text transcripts) for WER calculation.

Scoring recipe

# Calculate simulation fidelity
mssl = multi_scale_spectral_loss(real_noisy_audio, simulated_noisy_audio)

# Train ASR model on simulated/real data
asr_model = ConformerDualPathASR()
asr_model.train(simulated_audio, transcripts)

# Evaluate ASR on test set
predictions = asr_model.predict(test_noisy_audio)
wer = word_error_rate(predictions, test_transcripts)

Common pitfalls

• Using the full 57.4h dataset for training the distortion model instead of the specified <60s parallel subset.
• Evaluating ASR on clean test data instead of the specified real noisy test set.
• Not filtering training chunks by s2t ratio (0.4 ≤ s2t < 1.0) which excludes non-speech/silence.

Evidence (verbatim from paper)

To evaluate the simulation fidelity, testing clean speeches are input to the trained distortion model and the MSSL[14] is calculated between the real and simulated noisy data. ... performances in terms of WER will be compared.

Citation

@misc{guo2022dentddsp,
  title={DENT-DDSP: Data-efficient noisy speech generator using differentiable digital signal processors for explicit distortion modelling and noise-robust speech recognition},
  author={Guo et al. (2022)},
  year={2022},
  note={arXiv:2208.00987}
}

• arXiv: 2208.00987