Dgfnet Av Sep Eval

dgfnet-av-sep-eval

DGFNet: End-to-End Audio-Visual Source Separation Based on Dynamic Gating Fusion — Yu et al. (2025) (arXiv:2504.21366, 2025)

What this evaluates

Evaluates audio-visual models on their ability to separate target musical instrument sounds from mixed audio using synchronized video cues. It probes cross-modal feature alignment and dynamic fusion of audio and visual signals for source separation in complex environments.

Datasets

• MUSIC — total 685; splits: test (-1)
• MUSIC-21 — total 1365; splits: test (-1)

Metrics

• SDR (primary) — range: other
  • Signal-to-Distortion Ratio computed via mir_eval.separation.bss_eval_sources. Measures the ratio of target signal energy to total distortion energy (interference + artifacts + noise).
• SIR — range: other
  • Signal-to-Interference Ratio computed via mir_eval. Measures the ratio of target signal energy to interference energy from other sources.
• SAR — range: other
  • Signal-to-Artifacts Ratio computed via mir_eval. Measures the ratio of target signal energy to artifacts introduced by the separation algorithm.

Input / output format

Input: Synchronized audio and video clips. Audio is downsampled to 11kHz (~6s clips), processed via STFT (1022-point Hann window, hop 256) and log-frequency resampled to 256xT spectrogram. Video is sampled at 1 FPS, 3 frames randomly selected, with detected objects resized/cropped to 224x224.

Output: Separated audio spectrogram or time-frequency mask for the target instrument(s).

Scoring recipe

import mir_eval
import numpy as np

def compute_metrics(y_true, y_pred):
    # y_true, y_pred: (n_channels, n_samples) numpy arrays
    sdr, sir, sar, perm = mir_eval.separation.bss_eval_sources(
        y_true, y_pred, compute_permutation=True
    )
    return {
        'SDR': float(np.mean(sdr)),
        'SIR': float(np.mean(sir)),
        'SAR': float(np.mean(sar))
    }

Common pitfalls

• Missing video portions in the public MUSIC/MUSIC-21 datasets force reliance on reported results from Chen et al. (2023) for several baselines, which may limit direct reproducibility of the full comparison table.
• Audio preprocessing uses non-standard parameters for speech separation (11kHz downsampling, 1022-point STFT), so metrics are not directly comparable to standard audio benchmarks like WSJ0 or LibriSpeech.
• Visual feature extraction depends on external object detectors (Faster R-CNN or Detic); performance bottlenecks may stem from detection errors rather than the fusion module itself.

Evidence (verbatim from paper)

We used three common evaluation metrics: Signal-to-Distortion Ratio (SDR), Signal-to-Interference Ratio (SIR), and Signal-to-Artifacts Ratio (SAR), and evaluated the results using the mir_eval library (Raffel et al., [2014]). Higher values for all metrics indicate better performance. Regarding the SDR score, our method outperforms baseline model (Chen et al., [2023]) by 0.62 dB.

Citation

@misc{yu2025dgfnet,
  title={DGFNet: End-to-End Audio-Visual Source Separation Based on Dynamic Gating Fusion},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2504.21366}
}

• arXiv: 2504.21366