qhjqhj00 anisora-eval
AniSora: Exploring the Frontiers of Animation Video Generation in the Sora Era — Jiang et al. (2024) (arXiv:2412.10255, 2024)
What this evaluates
Evaluates the quality and controllability of AI-generated animation videos, specifically probing character consistency, style consistency, and distortion detection. It addresses the unique challenges of non-photorealistic content, exaggerated motion, and artistic coherence that standard video benchmarks often miss.
Datasets
- AniSora Benchmark — total 948; splits: test (948); repo https://github.com/bilibili/Index-anisora
Metrics
character consistency(primary) — range: [0, 1]- Measures how well the generated video maintains the visual identity and features of the target character across all frames. Specific mathematical formula not provided in the excerpt.
style consistency— range: [0, 1]- Evaluates the preservation of the target artistic style and non-photorealistic rendering quality throughout the video sequence. Specific mathematical formula not provided in the excerpt.
distortion detection— range: [0, 1]- Assesses the presence of visual artifacts, unnatural deformations, or temporal flickering in the generated animation. Specific mathematical formula not provided in the excerpt.
Input / output format
Input: Guide frames, text prompts, and motion masks for video generation tasks.
Output: Generated video clips.
Scoring recipe
# Not fully detailed in the provided text.
# Typically involves computing the six tailored metrics on the 948-video test set.
# score = evaluate_consistency(generated_video, gold_video)
# Returns average metric values across the benchmark for character consistency, style consistency, and distortion detection.
Common pitfalls
- Standard video generation metrics often fail to capture the specific requirements of non-photorealistic animation, such as exaggerated motion and artistic style preservation.
- Character consistency evaluation requires frame-level tracking that automated metrics may not accurately perform without specialized design tailored to animation.
Evidence (verbatim from paper)
a novel benchmark of 948 animation videos with six tailored evaluation metrics (character consistency, style consistency, distortion detection, etc.)—the first dedicated evaluation framework for animation video generation
Citation
@misc{jiang2024anisora,
title={AniSora: Exploring the Frontiers of Animation Video Generation in the Sora Era},
author={Jiang et al. (2024)},
year={2024},
note={arXiv:2412.10255}
}
- arXiv: 2412.10255