Videodpo Eval

videodpo-eval

VideoDPO: Omni-Preference Alignment for Video Diffusion Generation — Liu et al. (2024) (arXiv:2412.14167, 2024)

What this evaluates

Evaluates text-to-video diffusion models on visual quality and semantic alignment with input prompts. It measures intra-frame fidelity, aesthetic appeal, and inter-frame temporal consistency using automated benchmarks and human-preference predictors.

Datasets

• VBench — total ?; splits: test (-1)

Metrics

• VBench (primary) — range: percent
  • Assesses quality and semantic alignment across 16 hierarchical dimensions, providing fine-grained evaluation scores for video generation.
• HPS (V) — range: other
  • Predicts scores of human preference for generated videos, trained on large-scale human preference datasets.
• PickScore — range: other
  • Predicts scores of human preference for generated videos, trained on large-scale human preference datasets.

Input / output format

Input: Text prompt conditioning a video diffusion model.

Output: Generated video.

Scoring recipe

def evaluate_model(model, prompts):
    scores = {'VBench': [], 'HPS_V': [], 'PickScore': []}
    for prompt in prompts:
        videos = model.generate(prompt, num_samples=4)
        for v in videos:
            scores['VBench'].append(vbench_score(v))
            scores['HPS_V'].append(hpsv_score(v))
            scores['PickScore'].append(pickscore_score(v))
    return {k: sum(v)/len(v) for k, v in scores.items()}

Common pitfalls

• HPS(V) scores can be insensitive to certain models (e.g., CogVideo) due to low generation quality or early release dates.
• Metrics are fully automated and may not perfectly correlate with human subjective judgments of video quality or alignment.

Evidence (verbatim from paper)

To evaluate our method and the baselines, we use the following metrics: VBench, a widely recognized benchmark that assesses both quality and semantic alignment in video generation across 16 hierarchical dimensions, providing fine-grained evaluation. HPS (V) and PickScore are also included as metrics; both are trained on large-scale human preference datasets and are designed to predict scores of human preference for generated videos.

Citation

@misc{liu2024videodpo,
  title={VideoDPO: Omni-Preference Alignment for Video Diffusion Generation},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2412.14167}
}

• arXiv: 2412.14167