Ears Wham Eval

ears-wham-eval

EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation — Richter et al. (2024) (arXiv:2406.06185, 2024)

What this evaluates

Evaluates speech enhancement models by measuring how well they recover clean speech from noisy mixtures across a wide range of signal-to-noise ratios and speaker demographics. The benchmark covers both controlled training/validation splits and a blind test set with unseen speakers and noise.

Datasets

• EARS-WHAM — total ?; splits: train (32485), val (632), test (886), blind_test (743)

Metrics

• SI-SDR (primary) — range: dB
  • Scale-Invariant Signal-to-Distortion Ratio. Measures the ratio between the target signal energy and the residual noise/error energy, invariant to scaling.
• POLQA — range: [1, 5]
  • Perceptual Objective Listening Quality Analysis. ITU standard for predicting perceived speech quality based on psychoacoustic models.
• PESQ — range: [-0.5, 4.5]
  • Perceptual Evaluation of Speech Quality. ITU standard for predicting speech quality by comparing enhanced and reference signals.
• ESTOI — range: [0, 1]
  • Extended Short-Time Objective Intelligibility. Predicts speech intelligibility based on temporal envelope correlation in short-time segments.
• SIGMOS — range: [1, 5]
  • Speech Quality MOS. Deep learning-based predictor of mean opinion score for speech quality.
• DNSMOS — range: [1, 5]
  • Deep Noise Suppression MOS. Deep learning-based predictor of mean opinion score for speech quality and noise suppression.
• WER — range: percent
  • Word Error Rate. Percentage of words incorrectly recognized compared to the reference transcript.

Input / output format

Input: Noisy audio file (and clean reference audio for intrusive metrics).

Output: Enhanced audio file.

Scoring recipe

def compute_metrics(predictions, references):
    scores = {}
    scores['SI-SDR'] = compute_si_sdr(predictions, references)  # dB
    scores['POLQA'] = compute_polqa(predictions, references)    # 1-5
    scores['PESQ'] = compute_pesq(predictions, references)      # -0.5 to 4.5
    scores['ESTOI'] = compute_estoi(predictions, references)    # 0-1
    scores['SIGMOS'] = compute_sigmos(predictions)              # 1-5
    scores['DNSMOS'] = compute_dnsmos(predictions)              # 1-5
    scores['WER'] = compute_wer(predictions, references)        # percent
    return scores

Common pitfalls

• SNR is computed using LKFS loudness (ITU-R BS.1770) excluding silent regions, not raw RMS.
• Blind test set excludes ground truth and requires submission to an online evaluation server.
• Metrics are computed at specific maximum frequencies (e.g., SI-SDR at 24 kHz, WER at 8 kHz).

Evidence (verbatim from paper)

Table 4: Results on EARS-WHAM. Column groups are the method name, intrusive metrics, non-intrusive metrics, and WER. Below each metric is the maximum frequency taken into account for the assessment.

Citation

@misc{richter2024ears,
  title={EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation},
  author={Richter et al. (2024)},
  year={2024},
  note={arXiv:2406.06185}
}

• arXiv: 2406.06185