Magicanime Eval

magicanime-eval

MagicAnime: A Hierarchically Annotated, Multimodal and Multitasking Dataset with Benchmarks for Cartoon Animation Generation — Xu et al. (2025) (arXiv:2507.20368, 2025)

What this evaluates

Evaluates generative models on cartoon animation tasks including audio-driven facial animation, face reenactment, image-to-video generation, and frame interpolation. It probes a model's ability to produce stylized, temporally consistent video with accurate facial details and cross-modal alignment.

Datasets

• MagicAnime-Bench — total 400; splits: test (400)

Metrics

• VSR (primary) — range: [0, 1] or percent
  • Valid Sample Ratio: the proportion of generated results that are successfully produced without failure, measuring model adaptability to the benchmark.
• PSNR — range: dB
  • Peak Signal-to-Noise Ratio: standard pixel-level fidelity metric comparing generated frames to ground truth.
• SSIM — range: [0, 1]
  • Structural Similarity Index: measures perceptual similarity between generated and reference frames.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity: deep-learning-based perceptual distance metric where lower values indicate higher similarity.
• L1-loss — range: other
  • Least Absolute Deviations: mean absolute pixel-wise difference between generated and reference frames.

Input / output format

Input: Reference image or video paired with a driving modality (audio track or keypoints) for animation tasks; reference image or keyframes for image-to-video and frame interpolation tasks.

Output: Generated video clips.

Scoring recipe

valid_count = 0
for pred in predictions:
    if pred is successfully_generated:
        valid_count += 1
VSR = valid_count / len(predictions)

psnr_vals, ssim_vals, lpips_vals, l1_vals = [], [], [], []
for pred, gt in zip(predictions, ground_truth):
    psnr_vals.append(calculate_psnr(pred, gt))
    ssim_vals.append(calculate_ssim(pred, gt))
    lpips_vals.append(calculate_lpips(pred, gt))
    l1_vals.append(calculate_l1_loss(pred, gt))

avg_psnr = mean(psnr_vals)
avg_ssim = mean(ssim_vals)
avg_lpips = mean(lpips_vals)
avg_l1 = mean(l1_vals)

Common pitfalls

• VSR measures model success/adaptability rather than visual quality, so a low VSR does not directly indicate poor generation fidelity.
• The benchmark includes 12% non-humanoid samples to test generalizability, which many baseline models fail to handle correctly.
• The paper does not specify whether LPIPS and L1-loss are aggregated per-frame or per-video, which can affect reported values.

Evidence (verbatim from paper)

we included Valid Sample Ratio (VSR), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), and Least Absolute Deviations (L1-loss) as evaluation metrics. Among them, the VSR refers to the portion of results that can be successfully generated, which represents the adaptability of our benchmark to different models. Other quantitative metrics including PSNR, L1-loss, SSIM, and LPIPS were used to assess pixel-level and perceptual-level performance.

Citation

@misc{xu2025magicanime,
  title={MagicAnime: A Hierarchically Annotated, Multimodal and Multitasking Dataset with Benchmarks for Cartoon Animation Generation},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2507.20368}
}

• arXiv: 2507.20368