Ears Reverb Eval

ears-reverb-eval

EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation — Richter et al. (2024) (arXiv:2406.06185, 2024)

What this evaluates

Evaluates dereverberation models by measuring their ability to remove room acoustics effects from speech using real room impulse responses with RT60 up to 2 seconds. The protocol ensures fair comparison by normalizing loudness and removing direct-path delays before convolution.

Datasets

• EARS-Reverb — total ?; splits: train (32485), val (632), test (886), blind_test (743)

Metrics

• SI-SDR (primary) — range: dB
  • Scale-Invariant Signal-to-Distortion Ratio. Measures the ratio between the target signal energy and the residual noise/error energy, invariant to scaling.
• POLQA — range: [1, 5]
  • Perceptual Objective Listening Quality Analysis. ITU standard for predicting perceived speech quality based on psychoacoustic models.
• PESQ — range: [-0.5, 4.5]
  • Perceptual Evaluation of Speech Quality. ITU standard for predicting speech quality by comparing enhanced and reference signals.
• ESTOI — range: [0, 1]
  • Extended Short-Time Objective Intelligibility. Predicts speech intelligibility based on temporal envelope correlation in short-time segments.
• SIGMOS — range: [1, 5]
  • Speech Quality MOS. Deep learning-based predictor of mean opinion score for speech quality.
• DNSMOS — range: [1, 5]
  • Deep Noise Suppression MOS. Deep learning-based predictor of mean opinion score for speech quality and noise suppression.
• WER — range: percent
  • Word Error Rate. Percentage of words incorrectly recognized compared to the reference transcript.

Input / output format

Input: Reverberant audio file (and clean reference audio).

Output: Dereverberated audio file.

Scoring recipe

def compute_metrics(predictions, references):
    scores = {}
    scores['SI-SDR'] = compute_si_sdr(predictions, references)  # dB
    scores['POLQA'] = compute_polqa(predictions, references)    # 1-5
    scores['PESQ'] = compute_pesq(predictions, references)      # -0.5 to 4.5
    scores['ESTOI'] = compute_estoi(predictions, references)    # 0-1
    scores['SIGMOS'] = compute_sigmos(predictions)              # 1-5
    scores['DNSMOS'] = compute_dnsmos(predictions)              # 1-5
    scores['WER'] = compute_wer(predictions, references)        # percent
    return scores

Common pitfalls

• RIRs are truncated up to the highest amplitude index to remove direct-path delay before convolution.
• Loudness of reverberant speech is normalized to match clean speech using LKFS.
• Only RIRs with RT60 ≤ 2 s are used, and multi-channel recordings use a randomly selected channel.

Evidence (verbatim from paper)

Table 5: Results for the blind test. Column groups are the method name, intrusive metrics, non-intrusive metrics, and WER. Values indicate mean and standard deviation.

Citation

@misc{richter2024ears,
  title={EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation},
  author={Richter et al. (2024)},
  year={2024},
  note={arXiv:2406.06185}
}

• arXiv: 2406.06185