Vbench++ Eval

vbench++-eval

VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models — Huang et al. (2024) (arXiv:2411.13503, 2024)

What this evaluates

Evaluates the quality and trustworthiness of text-to-video and image-to-video generative models across 16 fine-grained dimensions, including spatial consistency, temporal dynamics, and subject identity. It measures how well automated scores align with human preferences and compares frame-wise generation capabilities against text-to-image baselines.

Datasets

• VBench++ — total ?; splits: test (-1); repo https://github.com/Vchitect/VBench

Metrics

• VBench score (primary) — range: [0, 1]
  • Automated score per dimension computed via a tailored evaluation method suite (e.g., CLIP similarity, optical flow, object tracking). Scores are linearly normalized to [0, 1] for cross-dimension comparison.
• Win Ratio — range: [0, 1]
  • Calculated from pairwise human comparisons: 1 for win, 0 for loss, 0.5 for tie. Total score divided by total pairwise comparisons.
• Spearman correlation — range: [-1, 1]
  • Correlation coefficient (ρ) between automated VBench scores and human preference win ratios to validate alignment.

Input / output format

Input: Text prompts (T2V) or reference image + text prompt (I2V) from the benchmark's prompt suites.

Output: Generated video clips, typically 2 seconds in length.

Scoring recipe

scores = {}
for dim in dimensions:
    video = generate(prompt_suite[dim])
    scores[dim] = run_automated_evaluator(video, prompt_suite[dim])
aggregate = mean(scores.values())
# Human alignment validation
win_ratio = (wins + 0.5 * ties) / total_comparisons
spearman_rho = spearmanr(automated_scores, human_win_ratios)

Common pitfalls

• Videos are standardized to 2 seconds, requiring frame sampling at 8 FPS for consistent evaluation.
• Empirical Max/Min baselines use retrieved WebVid-10M videos or Gaussian noise, not model outputs, which can mislead if interpreted as generation targets.
• Scores are linearly normalized per category for visualization, potentially masking absolute performance gaps.

Evidence (verbatim from paper)

For every dimension, we calculate the VBench scores using the evaluation method suite described in Section 3.1... Win Ratio. Given the human labels, we calculate the win ratio of each model. During pairwise comparisons, if a model’s video is selected as better, then the model scores 1 and the other model scores 0. If there is a tie, then both models score 0.5. For each model, the win ratio is calculated as the total score divided by the total number of pair-wise comparisons participated.

Citation

@misc{huang2024vbench++,
  title={VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2411.13503}
}

• arXiv: 2411.13503