qhjqhj00 vmbench-eval
VMBench: A Benchmark for Perception-Aligned Video Motion Generation — Ling et al. (2025) (arXiv:2503.10076, 2025)
What this evaluates
Evaluates how well text-to-video models generate videos that align with human perceptual preferences across five motion dimensions: object integrity, motion smoothness, commonsense adherence, perceptible amplitude, and temporal coherence.
Datasets
- MMPG-set — total 1050; splits: test (1050); repo https://github.com/GD-AIGC/VMBench
Metrics
Spearman correlation(primary) — range: percent- Spearman’s rank correlation coefficient ($\rho$) measures the monotonic relationship between two ranked variables. The paper reports $\rho \times 100$ to express alignment strength as a percentage.
Accuracy— range: percent- Percentage of video pairs where the metric's preference matches the human preference. Ties are explicitly excluded to prioritize unambiguous decisions.
Input / output format
Input: Text prompts (1,050 total) fed to text-to-video models to generate videos. For evaluation, videos are uniformly sampled at 2 frames per second and processed by rule-based algorithms or Multimodal Large Language Models (MLLMs).
Output: Per-dimension scores (1–5 Likert scale) from MLLMs or rule-based metrics, aggregated into PMM scores. Final outputs are Spearman correlation coefficients and alignment accuracy percentages against human expert ratings.
Scoring recipe
def compute_spearman(metric_scores, human_scores):
metric_ranks = rank(metric_scores)
human_ranks = rank(human_scores)
rho = spearmanr(metric_ranks, human_ranks).correlation
return rho * 100
def compute_accuracy(metric_scores, human_scores):
pairs = list(itertools.combinations(range(len(metric_scores)), 2))
matches = 0
total = 0
for i, j in pairs:
if human_scores[i] == human_scores[j]: continue
total += 1
if (metric_scores[i] > metric_scores[j]) == (human_scores[i] > human_scores[j]):
matches += 1
return (matches / total) * 100
Common pitfalls
- Ties in human preference pairs are explicitly excluded when computing alignment accuracy, which reduces the effective sample size and must be handled correctly.
- MLLM evaluation requires fixed frame sampling (2 fps) to manage computational limits; varying this rate changes the input distribution and breaks comparability.
- Rule-based metrics like RAFT and SSIM can yield negative correlations with human scores for certain dimensions (e.g., PAS), indicating counterintuitive alignment rather than better performance.
Evidence (verbatim from paper)
Spearman correlation is nonparametric and can effectively evaluate associations in datasets where variables may not follow a normal distribution. Unlike Pearson correlation, which captures linear relationships, Spearman correlation focuses on rank-based associations, making it more robust to outliers and suitable for ordinal data or scenarios with nonlinear dependencies. Alignment accuracy is quantified as the percentage of pairs where PMM preferences match human judgments, with ties excluded to prioritize unambiguous decisions.
Citation
@misc{ling2025vmbench,
title={VMBench: A Benchmark for Perception-Aligned Video Motion Generation},
author={Ling et al. (2025)},
year={2025},
note={arXiv:2503.10076}
}
- arXiv: 2503.10076