Videoscore Eval

videoscore-eval

VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation — He et al. (2024) (arXiv:2406.15252, 2024)

What this evaluates

Evaluates how well automatic video quality metrics correlate with human ratings across multiple dimensions such as visual quality, temporal consistency, and text alignment. It also measures pairwise preference accuracy to simulate human choice between generated videos.

Datasets

• VideoFeedback-test — total 760; splits: test (760)
• GenAI-Bench — total ?; splits: test (-1)
• VBench — total ?; splits: test (-1)
• EvalCrafter — total 2541; splits: test (2541)

Metrics

• Spearman's ρ (primary) — range: [-1, 1]
  • Rank-based correlation coefficient measuring the monotonic relationship between predicted scores and human reference scores across video dimensions.
• Pairwise preference accuracy — range: [0, 1]
  • Percentage of correct predictions when choosing which of two videos is preferred by humans, based on averaged aspect scores.

Input / output format

Input: Video (up to 128 frames) and corresponding text prompt.

Output: For scoring: aspect scores (1-4 integers or continuous values). For preference: binary choice between two videos.

Scoring recipe

def compute_spearman(pred, gold):
    return scipy.stats.spearmanr(pred, gold).correlation

def compute_pairwise_acc(pred_scores_A, pred_scores_B, human_pref_A):
    avg_A = np.mean(pred_scores_A, axis=1)
    avg_B = np.mean(pred_scores_B, axis=1)
    pred_pref_A = avg_A > avg_B
    return np.mean(pred_pref_A == human_pref_A)

Common pitfalls

• Discretizing continuous feature-based metrics to [1,4] using fixed thresholds before correlation calculation.
• Averaging five distinct aspect scores to derive a single preference score for pairwise comparison tasks.
• Excluding MLLM baselines that fail to follow the output format, potentially underestimating their true capability.

Evidence (verbatim from paper)

We take label prediction accuracy and Spearman’s $ ho$ in each dimension as evaluation indicators.

Citation

@misc{he2024videoscore,
  title={VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation},
  author={He et al. (2024)},
  year={2024},
  note={arXiv:2406.15252}
}

• arXiv: 2406.15252