Dualbench Eval

dualbench-eval

DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis — Tian et al. (2025) (arXiv:2507.10109, 2025)

What this evaluates

Evaluates a model's ability to generate synchronized background audio and intelligible speech from video input, measuring audio quality, distribution matching, and audio-video temporal alignment.

Datasets

• DualBench — total ?; splits: test (-1)
• VGGSound — total ?; splits: test (-1)

Metrics

• FAD↓ (primary) — range: other
  • Fréchet Audio Distance: measures the distance between feature distributions of generated and real audio using a pre-trained audio classifier. Lower is better.
• AV-Align↑ — range: [0, 1]
  • Audio-Video Alignment score: quantifies the temporal synchronization and acoustic harmony between generated audio and input video frames. Higher is better.
• FD↓ — range: other
  • Fréchet Distance: measures the distance between feature distributions of generated and real audio in a latent space. Lower is better.
• KLD↓ — range: other
  • Kullback-Leibler Divergence: measures the difference between the probability distributions of generated and real audio features. Lower is better.
• IS↑ — range: other
  • Inception Score: evaluates the quality and diversity of generated audio based on a classifier's confidence and entropy. Higher is better.

Input / output format

Input: Video frames processed via a CLIP ViT-B/32 visual encoder; optionally text prompts (though baselines use video-only input).

Output: Synchronized audio waveform containing both background sound and speech, sampled at 24kHz with 40 tokens per second.

Scoring recipe

def compute_metrics(generated_audios, reference_audios, videos):
    # Extract features using pre-trained audio/video encoders
    gen_feats = extract_features(generated_audios)
    ref_feats = extract_features(reference_audios)
    
    # Compute distribution distances
    fd = frechet_distance(gen_feats, ref_feats)
    fad = frechet_audio_distance(gen_feats, ref_feats)
    kld = kl_divergence(gen_feats, ref_feats)
    is_score = inception_score(gen_feats)
    
    # Compute alignment
    av_align = compute_audio_video_alignment(generated_audios, videos)
    
    return {'FD': fd, 'FAD': fad, 'KLD': kld, 'IS': is_score, 'AV-Align': av_align}

Common pitfalls

• Speech leaking into background audio tracks during generation or evaluation
• Mismatched frame rates between video features and audio tokens causing misalignment
• Using text prompts during inference for video-to-audio baselines, violating the video-only setting

Evidence (verbatim from paper)

Table 2: Objective evaluation of generated audio on the VGGSound and DualBench test sets. The best and the second best result is shown in bold and by underlined.

| Model | VGGSound | ... | FD↓ | FAD↓ | KLD↓ | IS↑ | AV-Align↑ |

Citation

@misc{tian2025dualdub,
  title={DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis},
  author={Tian et al. (2025)},
  year={2025},
  note={arXiv:2507.10109}
}

• arXiv: 2507.10109