qhjqhj00 opens2v-eval
OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation — Yuan et al. (2025) (arXiv:2505.20292, 2025)
What this evaluates
Evaluates Subject-to-Video (S2V) generation models on their ability to maintain subject identity consistency, produce natural temporal dynamics, and align with text prompts. It covers open-domain, human-specific, and single-subject scenarios to expose common failure modes like copy-paste artifacts and fidelity degradation.
Datasets
- OpenS2V-Eval — total 180; splits: test (180); repo https://github.com/PKU-YuanGroup/OpenS2V-Nexus
Metrics
NexusScore(primary) — range: percent- Measures subject consistency and identity fidelity between the reference image and the generated video.
NaturalScore(primary) — range: percent- Evaluates temporal naturalness and detects copy-paste artifacts or unnatural pose/lighting replication.
GmeScore(primary) — range: percent- Assesses text relevance and alignment between the input prompt and the generated video content.
Aesthetics— range: percent- Measures overall visual quality using an improved aesthetic predictor.
Motion— range: percent- Quantifies motion amplitude and dynamics in the generated video.
FaceSim— range: percent- Computes face similarity between the reference subject and the generated video using FaceSim-Cur.
Total Score— range: percent- A normalized weighted sum of the six dimension scores (Aesthetics, Motion, FaceSim, GmeScore, NexusScore, NaturalScore).
Input / output format
Input: Reference subject image(s) and a text prompt specifying the desired video content, motion, and subject type.
Output: A generated video clip (typically 720p resolution, e.g., 81 frames × 720 × 1280 pixels).
Scoring recipe
def evaluate(generated_video, ref_image, prompt):
aesthetics = run_aesthetic_predictor(generated_video)
motion = run_motion_score(generated_video)
facesim = run_facesim_cur(ref_image, generated_video)
gme = run_gme_score(prompt, generated_video)
nexus = run_nexus_score(ref_image, generated_video)
natural = run_natural_score(generated_video)
total = normalize_weighted_sum([aesthetics, motion, facesim, gme, nexus, natural])
return {'Aesthetics': aesthetics, 'Motion': motion, 'FaceSim': facesim, 'GmeScore': gme, 'NexusScore': nexus, 'NaturalScore': natural, 'Total Score': total}
Common pitfalls
- Models frequently exhibit 'copy-paste' issues, incorrectly replicating expressions, lighting, or poses from the reference image into the video.
- Subject fidelity tends to degrade over time or when multiple subjects are present in the prompt.
- Relying solely on coarse frame-level similarity metrics misses critical identity fidelity and temporal consistency.
Evidence (verbatim from paper)
Pika [pika] achieves the highest GmeScore, indicating that the generated videos are better aligned with the provided instructions. Kling [KeLing], on the other hand, produces videos with higher fidelity and realism, securing the highest NexusScore and NaturalScore.
Citation
@misc{yuan2025opens2vnexus,
title={OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation},
author={Yuan et al. (2025)},
year={2025},
note={arXiv:2505.20292}
}
- arXiv: 2505.20292