Msvbench Eval

msvbench-eval

MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation — Haoyuan Shi et al. (2026) (arXiv:2602.23969, 2026)

What this evaluates

Evaluates multi-shot video generation models on narrative coherence, cross-shot consistency, visual fidelity, and motion quality. It probes whether models can maintain character and scene identity across sequential shots and adhere to physical laws, rather than merely generating isolated visual interpolations.

Datasets

• MSVBench — total ?; splits: test (-1)

Metrics

• Spearman’s ρ (primary) — range: percent
  • Spearman rank correlation coefficient measuring the monotonic relationship between automated benchmark scores and human Mean Opinion Scores (MOS).
• Dover Score — range: [0, 1]
  • Perceptual quality metric assessing visual fidelity and aesthetic appeal of generated video frames.
• VQAScore — range: [0, 100]
  • Video-Question Answering Score evaluating semantic alignment between the generated video and the story script.
• Kendall’s τ — range: percent
  • Kendall rank correlation coefficient measuring ordinal association between model rankings and human preferences.

Input / output format

Input: Hierarchical story scripts and per-shot reference images/keyframes provided to the generation model.

Output: Multi-shot video sequences generated by the model, evaluated against gold scripts and reference images.

Scoring recipe

def evaluate_video(video, script, ref_images):
    scores = {}
    # Perceptual & Consistency (Expert Models)
    scores['visual'] = compute_dover_musiq(video, ref_images)
    scores['consistency'] = compute_cross_shot_consistency(video) # Face, Char, Back, Cloth, Size
    # Semantic Alignment (LMMs)
    scores['alignment'] = compute_vqa_score(video, script)
    scores['motion'] = compute_motion_metrics(video) # AR, AS, Cam, PhyP, PhyI
    # Human Alignment
    human_mos = get_human_ratings(video)
    scores['spearman_rho'] = spearmanr(scores['alignment'], human_mos)
    scores['kendall_tau'] = kendalltau(scores['alignment'], human_mos)
    return scores

Common pitfalls

• Models often act as local visual interpolators, failing to maintain character or attribute consistency across shots.
• Reference images constrain generation, often reducing physical plausibility compared to text-only conditioning.
• High motion intensity frequently degrades physical interaction accuracy and content preservation due to architectural trade-offs.

Evidence (verbatim from paper)

The metrics are categorized into four dimensions: Visual Quality (Dov.: Dover Score, Mus.: MusIQ Score, V.A.C: Visual Attribute Consistency, S.C.: Style Consistency); Story Video Alignment (VQA: VQAScore, Det.: Detection & Count Score, S.P.A: Shot Perspective Alignment, S.S.P: State Shift & Persistence, S.V.C: Story Video Consistency); Video Consistency (Fac.: Face Consistency, Cha.: Character Consistency, Bac.: Background Consistency, Clo.: Clothes & Color Consistency, Siz.: Relative Size Consistency); and Motion Quality (A.R.: Action Recognition, A.S.: Action Strength, Cam.: Camera Control, Phy.P: Physical Plausibility, Phy.I: Physical Interaction Accuracy).

Citation

@misc{shi2026msvbench,
  title={MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation},
  author={Haoyuan Shi et al. (2026)},
  year={2026},
  note={arXiv:2602.23969}
}

• arXiv: 2602.23969