Fuss Sound Separation Eval

fuss-sound-separation-eval

What's All the FUSS About Free Universal Sound Separation Data? — Wisdom et al. (2020) (arXiv:2011.00803, 2020)

What this evaluates

Evaluates audio source separation models on mixed reverberant or dry recordings containing 1–4 active sources. It measures reconstruction fidelity and source-counting accuracy across varying mixture complexities.

Datasets

• FUSS — total ?; splits: val (-1), test (-1)

Metrics

• SI-SNR (primary) — range: dB
  • Scale-invariant signal-to-noise ratio computed via cosine similarity: SI-SNR(y, ŷ) ≈ 10 log10((ρ²(y, ŷ) + ε) / (1 - ρ²(y, ŷ) + ε)), where ρ is the cosine similarity between target y and estimate ŷ, and ε=10⁻⁸. Estimates are first aligned to references via permutation matching, and pairs with inactive references or estimates >20 dB below the quietest active reference are discarded.

Input / output format

Input: Mixed audio waveforms (dry or reverberant) containing 1 to 4 active sound sources.

Output: Separated audio waveforms corresponding to each estimated source.

Scoring recipe

def compute_sisnr(y, y_hat, eps=1e-8):
    # 1. Align estimates to references by maximizing SI-SNR over permutations
    # 2. Discard pairs where y is all-zeros or power(y_hat) < power(quietest_nonzero_ref) - 20dB
    rho = np.dot(y, y_hat) / (np.linalg.norm(y) * np.linalg.norm(y_hat) + eps)
    rho_sq = rho ** 2
    sisnr = 10 * np.log10((rho_sq + eps) / (1 - rho_sq + eps))
    return sisnr

Common pitfalls

• Using the original scale-based SI-SNR formula instead of the cosine-similarity variant yields overly optimistic scores, particularly for under-separation.
• Failing to align estimates to references via permutation matching before computing scores.
• Not filtering out estimate-reference pairs where the reference is inactive or the estimate power is >20 dB below the quietest active reference.

Evidence (verbatim from paper)

To evaluate performance, we use scale-invariant signal-to-noise ratio (SI-SNR)... To correct for this, we use an alternate formulation, SI-SNR(y,ŷ) = 10 log10(ρ²(y,ŷ)/(1-ρ²(y,ŷ))) ≈ 10 log10((ρ²(y,ŷ)+ε)/(1-ρ²(y,ŷ)+ε)), where ρ(y,ŷ) = yᵀŷ/(||y||||ŷ||) ≈ yᵀŷ/(||y||||ŷ||+ε) is the cosine similarity between y and ŷ, with stabilizer ε. In our experiments we use ε=10⁻⁸.

Citation

@misc{wisdom2020fuss,
  title={What's All the FUSS About Free Universal Sound Separation Data?},
  author={Wisdom et al. (2020)},
  year={2020},
  note={arXiv:2011.00803}
}

• arXiv: 2011.00803