Vabench Eval

vabench-eval

VABench: A Comprehensive Benchmark for Audio-Video Generation — Hua et al. (2025) (arXiv:2512.09299, 2025)

What this evaluates

Evaluates the quality, cross-modal consistency, synchronization, and spatial audio rendering of text-to-audio-video and image-to-audio-video generation models. It probes physical plausibility, emotional expressiveness, and stereo separation across seven real-world sound categories.

Datasets

• VABench — total ?; splits: test (-1); repo https://github.com/tanABCC/VABench

Metrics

• Audio-Visual Align (primary) — range: [0, 1]
  • Cross-modal semantic alignment score between generated audio and video, assessed via expert-based and MLLM-driven evaluation.
• Lip-Sync — range: [0, 1]
  • Synchronization accuracy for speech and lip movements, measured via expert/MLLM assessment.
• Audio Realism — range: [0, 1]
  • Fidelity and naturalness of the generated audio track.
• Video Realism — range: [0, 1]
  • Fidelity and naturalness of the generated video sequence.
• Audio QA — range: [0, 1]
  • Fine-grained semantic understanding and question-answering capability on the generated audio.
• Vision QA — range: [0, 1]
  • Fine-grained semantic understanding and question-answering capability on the generated video.
• Phase Coherence — range: [0, 1]
  • Normalized acoustic metric measuring left-right channel phase consistency across low, mid, and high frequencies.
• Mono Compatibility — range: [0, 1]
  • Normalized acoustic metric evaluating how well the stereo mix collapses to mono without phase cancellation artifacts.
• Soundstage Width — range: [0, 1]
  • Normalized acoustic metric quantifying the perceived spatial width of the audio field.
• Transient Synchronization — range: [0, 1]
  • Normalized acoustic metric measuring temporal alignment of sharp sound events across channels.
• Level Stability — range: [0, 1]
  • Normalized acoustic metric assessing consistent amplitude levels across the stereo track.
• Envelope Correlation — range: [0, 1]
  • Normalized acoustic metric evaluating the similarity of amplitude envelope shapes between left and right channels.
• Imaging Stability — range: [0, 1]
  • Normalized acoustic metric measuring the steadiness of localized sound sources in the stereo field.

Input / output format

Input: Text prompts (for T2AV) or reference images (for I2AV), optionally with audio prompts for stereo tasks. Models generate synchronized video and audio outputs.

Output: Synchronized video (720P, default frame rate/duration) and audio (48kHz stereo tracks) files.

Scoring recipe

def compute_vabench_scores(predictions, gold_prompts):
    scores = {}
    for model, (vid, aud) in predictions.items():
        scores[model] = {
            'Audio-Visual Align': eval_cross_modal(vid, aud),
            'Lip-Sync': eval_lip_sync(vid, aud),
            'Audio Realism': eval_realism(aud),
            'Video Realism': eval_realism(vid),
            'Audio QA': eval_qa(aud, gold_prompts),
            'Vision QA': eval_qa(vid, gold_prompts),
            **{m: eval_acoustic_metric(aud, m) for m in STEREO_METRICS}
        }
    return scores
# Human validation: 6 evaluators rate subset on 1-5 scale. Compute pairwise win rates (Win=1, Loss=0, Tie=0.5) for humans and benchmark scores. Calculate Pearson correlation between the two win-rate vectors.

Common pitfalls

• Confusing end-to-end audio-video models with decoupled video+audio pipelines, which have fundamentally different training objectives and performance characteristics.
• Assuming high stereo acoustic metric scores imply semantic spatial awareness; the paper notes models often produce technical stereo separation without corresponding visual or semantic grounding.
• Evaluating across only a single sound category; performance varies significantly across the seven categories (e.g., strong on Music/Animals, weak on Human Sounds).

Evidence (verbatim from paper)

To validate VABench’s alignment with human senses, we conducted a pilot user study. Balancing evaluation fidelity and cost, we had six professional evaluators rate a representative subset of videos from three models (Veo3, Seedance+MMAudio, Wan2.2+ThinkSound) on a 1–5 scale. The evaluation focused on three key dimensions—semantics, synchronization, and realism—which directly correspond to aggregated benchmark metrics (Semantics: Text-Video Align, Text-Audio Align, Audio-Visual Align; Synchronization: Desync; Realism: Audio Realism, Video Realism).

Citation

@misc{hua2025vabench,
  title={VABench: A Comprehensive Benchmark for Audio-Video Generation},
  author={Hua et al. (2025)},
  year={2025},
  note={arXiv:2512.09299}
}

• arXiv: 2512.09299