Openvid 1m Eval

openvid-1m-eval

OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation — Nan et al. (2024) (arXiv:2407.02371, 2024)

What this evaluates

Evaluates text-to-video generation models on visual aesthetics, technical quality, text-video alignment, and temporal consistency using a standardized set of 700 prompts.

Datasets

• Liu et al. (2023b) Benchmark — total 700; splits: test (700)

Metrics

• VQAA (primary) — range: percent
  • Aesthetic score measuring the visual appeal and artistic quality of generated videos.
• VQAT — range: percent
  • Technical score assessing video clarity, resolution fidelity, and absence of artifacts.
• Blip_bleu — range: other
  • Text-text consistency metric computed by comparing the input prompt with a caption generated from the video using BLIP.
• SD_score — range: percent
  • Image-video consistency metric measuring alignment between the input text prompt and the generated video frames.
• Clip_temp_score — range: percent
  • Semantic consistency metric evaluating temporal coherence across video frames using CLIP embeddings.
• Warping_error — range: other
  • Temporal consistency metric quantifying frame-to-frame motion smoothness via optical flow warping error.

Input / output format

Input: Text prompt (selected from the 700-prompt benchmark set).

Output: Generated video clip (typically 16 frames sampled at 3-frame intervals, at specified resolutions such as 256x256, 512x512, or 1024x1024).

Scoring recipe

def compute_metrics(predictions, gold_prompts):
    scores = {}
    for prompt, video in zip(gold_prompts, predictions):
        scores['VQAA'] += aesthetic_score(video)
        scores['VQAT'] += technical_score(video)
        scores['Blip_bleu'] += blip_bleu(prompt, video)
        scores['SD_score'] += sd_alignment(prompt, video)
        scores['Clip_temp_score'] += clip_temp_consistency(video)
        scores['Warping_error'] += warping_error(video)
    n = len(predictions)
    return {k: v/n for k, v in scores.items()}

Common pitfalls

• Resolution drastically impacts scores; models must be evaluated at identical resolutions for fair comparison.
• Super-resolution is sometimes applied to lower-resolution datasets to match high-resolution benchmarks, which can artificially inflate quality metrics.
• Training steps and GPU hours must be controlled, as convergence differences can be mistaken for inherent model capability.

Evidence (verbatim from paper)

We evaluate our model on public benchmark in Liu et al. (2023b), which evaluates text-to-video generation model based on visual quality, text-video alignment and temporal consistency. Specifically, we adopt aesthetic score (VQAA) and technical score(VQAT) for video quality assessment. We evaluate the alignment of input text and generated video in two aspects, including image-video consistency (SD_score) and text-text consistency (Blip_bleu). We also evaluate temporal consistency of generated video with warping error and semantic consistency (Clip_temp_score).

Citation

@misc{nan2024openvid1m,
  title={OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation},
  author={Nan et al. (2024)},
  year={2024},
  note={arXiv:2407.02371}
}

• arXiv: 2407.02371