Whamr Eval

whamr-eval

WHAMR!: Noisy and Reverberant Single-Channel Speech Separation — Maciejewski et al. (2019) (arXiv:1910.10279, 2019)

What this evaluates

Evaluates single-channel speech separation and enhancement (denoising/dereverberation) capabilities under realistic noisy and reverberant conditions using synthetically generated reverberant mixtures.

Datasets

• WHAMR! — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• SI-SDR (primary) — range: other
  • Scale-invariant signal-to-distortion ratio. Formula: 10log10(||αs||² / ||αs - ŝ||²), where α = ⟨ŝ, s⟩ / ||s||². It is scale-invariant and used for both training and evaluation.

Input / output format

Input: 4-second audio segments of single-channel mixtures containing speech sources, noise, and/or reverberation.

Output: Estimated waveform for each target source (or enhanced source).

Scoring recipe

def compute_si_sdr(target, estimate):
    alpha = np.dot(estimate, target) / np.dot(target, target)
    scaled_target = alpha * target
    noise = scaled_target - estimate
    s_power = np.dot(scaled_target, scaled_target)
    n_power = np.dot(noise, noise)
    return 10 * np.log10(s_power / n_power)

Common pitfalls

• Outputs from cascaded models may have dynamic range mismatches due to the scale-invariant loss, requiring explicit rescaling before cascading or evaluation.
• SI-SDR is undefined for silent sources, so training on large subsets with sparse speech can cause instability; the authors train on a smaller subset to avoid silent segments.

Evidence (verbatim from paper)

We train all networks using permutation invariant training with the scale-invariant signal-to-distortion ratio (SI-SDR, also referred to as SI-SNR) waveform-level training objective. SI-SDR is also the evaluation metric and allows for end-to-end joint training of cascaded enhancement and separation models: SI-SDR = 10log10(||αs||² / ||αs - ŝ||²), α = ⟨ŝ, s⟩ / ||s||².

Citation

@misc{maciejewski2019whamr,
  title={WHAMR!: Noisy and Reverberant Single-Channel Speech Separation},
  author={Maciejewski et al. (2019)},
  year={2019},
  note={arXiv:1910.10279}
}

• arXiv: 1910.10279