Biodenoising Eval

biodenoising-eval

Biodenoising: Animal Vocalization Denoising without Access to Clean Data — Miron et al. (2024) (arXiv:2410.03427, 2024)

What this evaluates

Evaluates the ability of audio denoising models to remove background noise from animal vocalization recordings without access to clean reference data during training. It measures how well models generalize across diverse species and environments using synthetic mixtures and a held-out benchmark set.

Datasets

• Biodenoising benchmark set — total 62; splits: test (62); repo https://github.com/earthspecies/biodenoising

Metrics

• SI-SDR (primary) — range: other
  • Scale-Invariant Signal-to-Distortion Ratio in dB. It computes the ratio between the target signal power and the distortion power (difference between scaled target and prediction), invariant to amplitude scaling. Also reports SI-SDRi, the improvement over the noisy mixture baseline.

Input / output format

Input: 4-second noisy audio waveforms sampled at 16 kHz, containing animal vocalizations mixed with background noise.

Output: 4-second denoised audio waveforms sampled at 16 kHz.

Scoring recipe

def compute_si_sdr(pred, target):
    target_power = np.sum(target**2)
    pred_scaled = (np.sum(pred * target) / target_power) * target
    noise = pred_scaled - pred
    noise_power = np.sum(noise**2)
    return 10 * np.log10(target_power / noise_power)

# Per test excerpt, average SI-SDR across seeds 0-9
excerpt_scores = [compute_si_sdr(pred_seed, target) for seed in range(10)]
excerpt_mean = np.mean(excerpt_scores)

# Aggregate across 62 test files
all_excerpts = [excerpt_mean for _ in range(62)]
report_median = np.median(all_excerpts)
report_mad = np.median(np.abs(all_excerpts - report_median))

Common pitfalls

• The test set is small (62 files), so the paper explicitly reports the median and median absolute deviation rather than the mean to avoid skew from outliers.
• Models must be evaluated across 10 random seeds (0-9) for the overall comparison, and SI-SDR scores must be averaged across seeds per excerpt before computing the final median.
• SI-SDRi is calculated as the difference between the denoised SI-SDR and the noisy mixture SI-SDR, not as a standalone metric.

Evidence (verbatim from paper)

Similarly to speech enhancement, we evaluate our model using an SNR metric, the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) [48] in dB units on the benchmarking set in Section III-B. In addition, we compute the improvement of this metric over the noisy mixtures SI-SDRi. The metrics are averaged for the full excerpts as follows. For each test excerpt we take the means of the metrics across the seeds. We report the median and the median absolute deviation across 62 files in the test set.

Citation

@misc{miron2024biodenoising,
  title={Biodenoising: Animal Vocalization Denoising without Access to Clean Data},
  author={Miron et al. (2024)},
  year={2024},
  note={arXiv:2410.03427}
}

• arXiv: 2410.03427