Neural Beamforming Adaptation Eval

neural-beamforming-adaptation-eval

Run-Time Adaptation of Neural Beamforming for Robust Speech Dereverberation and Denoising — Fujita et al. (2024) (arXiv:2410.22805, 2024)

What this evaluates

Evaluates the robustness of a run-time adapted neural beamforming system for joint speech dereverberation and denoising under varying acoustic conditions, including different numbers of speakers, reverberation times, and SNRs.

Datasets

• Simulated Librispeech+DEMAND — total ?; splits: train (36000), val (3600), test (210)

Metrics

• WER (primary) — range: [0, 1] or percent
  • Word Error Rate: edit distance between predicted and reference transcriptions divided by reference length.
• SDR — range: dB
  • Signal-to-Distortion Ratio: ratio of target signal energy to distortion energy (noise, artifacts, interference) in decibels.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility: temporal correlation-based metric measuring speech envelope similarity between clean and enhanced signals.
• PESQ — range: [-0.5, 4.5]
  • Perceptual Evaluation of Speech Quality: ITU-T P.862 standard mapping to a MOS-like score based on perceptual speech quality.
• SRMR — range: dB
  • Speech-to-Reverberation Modulation Energy Ratio: ratio of modulation energy in speech-dominant bands to reverberation-dominant bands.

Input / output format

Input: 7-channel STFT coefficients (window=1024, hop=256) of the mixture signal, concatenated with log magnitude, inter-channel phase differences, and delay-sum beamforming output. Target DOA is provided as auxiliary input.

Output: Time-frequency (TF) mask for weighted power minimization distortionless response (WPD) beamforming, plus adapted filter parameters for WPE and FastMNMF.

Scoring recipe

def compute_metrics(predictions, gold):
    wer = compute_wer(predictions.transcription, gold.transcription)
    sdr = compute_sdr(predictions.waveform, gold.waveform)
    stoi = compute_stoi(predictions.waveform, gold.waveform)
    pesq = compute_pesq(predictions.waveform, gold.waveform)
    srmer = compute_srmer(predictions.waveform, gold.waveform)
    return {'WER': wer, 'SDR': sdr, 'STOI': stoi, 'PESQ': pesq, 'SRMR': srmer}

Common pitfalls

• Evaluation uses only the second half (~4 minutes) of each 8-minute test recording; the first half is strictly reserved for fine-tuning the mask estimator.
• No ground-truth speech is available during run-time adaptation; the system relies on pseudo-ground-truth estimates from FastMNMF.
• Test noise is strictly from DEMAND's 'PSTATION' environment, while training/validation noise explicitly excludes it to prevent data leakage.

Evidence (verbatim from paper)

The evaluation compared different amounts of fine-tuning data (30, 60, 120, and 240 seconds) against several key metrics. These metrics include word error rate (WER), signal-to-distortion ratio (SDR), short-time objective intelligibility (STOI), perceptual evaluation of speech quality (PESQ), and speech-to-reverberation modulation energy ratio (SRMR).

Citation

@misc{fujita2024runtime,
  title={Run-Time Adaptation of Neural Beamforming for Robust Speech Dereverberation and Denoising},
  author={Fujita et al. (2024)},
  year={2024},
  note={arXiv:2410.22805}
}

• arXiv: 2410.22805