vbench-eval
Taming Diffusion Transformer for Efficient Mobile Video Generation in Seconds — Wu et al. (2025) (arXiv:2507.13343, 2025)
What this evaluates
Evaluates the visual quality, semantic alignment, temporal consistency, and aesthetic fidelity of text-to-video generation models. It probes the model's ability to produce coherent, high-fidelity videos that match textual prompts across multiple perceptual and technical dimensions.
Datasets
- VBench — total 1000; splits: test (1000)
Metrics
VBench Total Score (primary) — range: percent
- Composite score aggregating multiple sub-dimensions including Quality, Semantic, Aesthetics, Imaging, Object Class, Scene, Consistency, and Flickering. Computed by generating 5 videos per prompt across a 1K prompt set and averaging the VBench evaluator outputs.
Input / output format
Input: Text prompt (encoded via T5 for server evaluation, CLIP for mobile evaluation)
Output: Video sequence (121 frames at 576x1024 for server; 49 frames at 384x512 for mobile), saved at 24 FPS for 5 seconds
Scoring recipe
prompts = load_vbench_1k_prompts()
all_scores = []
for prompt in prompts:
videos = [generate_video(prompt, steps=4, cfg=None, seed=s) for s in range(5)]
scores = vbench_evaluate(videos)
all_scores.append(scores)
final_metrics = {k: mean([s[k] for s in all_scores]) for k in all_scores[0].keys()}
return final_metrics
Common pitfalls
- Using classifier-free guidance (CFG) during evaluation, which contradicts the paper's 4-step distillation setup that explicitly runs without CFG.
- Evaluating with different video resolutions or frame counts than specified (576x1024x121 for server, 384x512x49 for mobile), as VBench scores are sensitive to resolution and temporal length.
- Averaging scores over fewer than 5 videos per prompt or using a different prompt subset than the official 1K VBench set, which can cause high variance (authors note ΔVBench score < ±0.2).
Evidence (verbatim from paper)
Our models are evaluated following the standard Vbench*(Huang et al., [2024])* setting, that is, we generate 5 videos for each prompt, and test the scores over the 1K prompt set. Both server and mobile-deployed models are step-distilled and evaluated with 4-step generation. The server model generates 121-frame horizontal videos at a resolution of 576 imes 1024, without classifier-free guidance.
Citation
@misc{wu2025taming,
title={Taming Diffusion Transformer for Efficient Mobile Video Generation in Seconds},
author={Wu et al. (2025)},
year={2025},
note={arXiv:2507.13343}
}
1---2name: vbench-eval3description: Evaluates the visual quality, semantic alignment, temporal consistency, and aesthetic fidelity of text-to-video generation models. It probes the model's ability to produce coherent, high-fidelity videos that match textual prompts across multiple perceptual and technical dimensions. Use when the user wants to benchmark on VBench, or asks about evaluating this task. Reports VBench Total Score.4---56# vbench-eval78> Taming Diffusion Transformer for Efficient Mobile Video Generation in Seconds — Wu et al. (2025) (arXiv:2507.13343, 2025)910## What this evaluates1112Evaluates the visual quality, semantic alignment, temporal consistency, and aesthetic fidelity of text-to-video generation models. It probes the model's ability to produce coherent, high-fidelity videos that match textual prompts across multiple perceptual and technical dimensions.1314## Datasets1516- **VBench** — total 1000; splits: test (1000)1718## Metrics1920- `VBench Total Score` **(primary)** — range: percent21 - Composite score aggregating multiple sub-dimensions including Quality, Semantic, Aesthetics, Imaging, Object Class, Scene, Consistency, and Flickering. Computed by generating 5 videos per prompt across a 1K prompt set and averaging the VBench evaluator outputs.2223## Input / output format2425**Input**: Text prompt (encoded via T5 for server evaluation, CLIP for mobile evaluation)2627**Output**: Video sequence (121 frames at 576x1024 for server; 49 frames at 384x512 for mobile), saved at 24 FPS for 5 seconds2829## Scoring recipe3031```python32prompts = load_vbench_1k_prompts()33all_scores = []34for prompt in prompts:35 videos = [generate_video(prompt, steps=4, cfg=None, seed=s) for s in range(5)]36 scores = vbench_evaluate(videos)37 all_scores.append(scores)38final_metrics = {k: mean([s[k] for s in all_scores]) for k in all_scores[0].keys()}39return final_metrics40```4142## Common pitfalls4344- Using classifier-free guidance (CFG) during evaluation, which contradicts the paper's 4-step distillation setup that explicitly runs without CFG.45- Evaluating with different video resolutions or frame counts than specified (576x1024x121 for server, 384x512x49 for mobile), as VBench scores are sensitive to resolution and temporal length.46- Averaging scores over fewer than 5 videos per prompt or using a different prompt subset than the official 1K VBench set, which can cause high variance (authors note ΔVBench score < ±0.2).4748## Evidence (verbatim from paper)4950> Our models are evaluated following the standard Vbench*(Huang et al., [2024])* setting, that is, we generate 5 videos for each prompt, and test the scores over the 1K prompt set. Both server and mobile-deployed models are step-distilled and evaluated with 4-step generation. The server model generates 121-frame horizontal videos at a resolution of 576 imes 1024, without classifier-free guidance.5152## Citation5354```bibtex55@misc{wu2025taming,56 title={Taming Diffusion Transformer for Efficient Mobile Video Generation in Seconds},57 author={Wu et al. (2025)},58 year={2025},59 note={arXiv:2507.13343}60}61```6263- arXiv: 2507.13343
qhjqhj00