Aigcbench Eval

aigcbench-eval

AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI — Fan et al. (2024) (arXiv:2401.01651, 2024)

What this evaluates

Evaluates the performance of image-to-video (I2V) generation models across multiple quality and alignment dimensions. It probes how well models preserve input image fidelity, generate coherent motion, align with text prompts, maintain temporal consistency, and produce high-quality video output.

Datasets

• AIGCBench Dataset — total ?; splits: test (-1); repo https://github.com/BenchCouncil/AIGCBench

Metrics

• flow-square-mean — range: [0, 1]
  • Measures optical flow magnitude to distinguish camera movement from subject motion. Higher values indicate more camera-centric motion.
• GenVideo-RefVideo CLIP — range: [0, 1]
  • Computes CLIP similarity between the generated video and the reference image/text to evaluate semantic and visual alignment.
• GenVideo-RefVideo SSIM — range: [0, 1]
  • Calculates Structural Similarity Index between the generated video and a reference video to measure pixel-level structural fidelity.
• image fidelity — range: percent
  • Assesses how well the generated video preserves the spatial structure and details of the input image.
• motion effects — range: percent
  • Evaluates the presence and quality of local and subject movement in the generated video.
• text-video alignment — range: percent
  • Measures the degree to which the generated video content matches the provided text prompt.
• temporal consistency — range: percent
  • Assesses the stability and coherence of visual content across consecutive frames.
• video quality (primary) — range: percent
  • Composite or standalone metric evaluating overall generation quality, including aesthetics and clarity.
• aesthetics — range: percent
  • Evaluates the visual appeal and artistic quality of the generated video frames.
• clarity — range: percent
  • Measures the sharpness and lack of blur or artifacts in the generated video.

Input / output format

Input: A reference image and a text prompt describing the desired video content.

Output: A generated video sequence (typically up to 96 frames).

Scoring recipe

def score(predictions, gold):
    pred_video = predictions['video']
    ref_image = gold['image']
    text_prompt = gold['prompt']
    ref_video = gold.get('ref_video')
    return {
        'flow-square-mean': compute_flow_magnitude(pred_video),
        'GenVideo-RefVideo CLIP': clip_similarity(pred_video, ref_image, text_prompt),
        'GenVideo-RefVideo SSIM': ssim(pred_video, ref_video) if ref_video else 0,
        'image fidelity': assess_fidelity(pred_video, ref_image),
        'motion effects': assess_motion(pred_video),
        'text-video alignment': clip_similarity(pred_video, ref_image, text_prompt),
        'temporal consistency': assess_consistency(pred_video),
        'video quality': assess_quality(pred_video),
        'aesthetics': assess_aesthetics(pred_video),
        'clarity': assess_clarity(pred_video)
    }

Common pitfalls

• Existing automated metrics struggle to capture fine-grained textual changes, leading to misalignment between metric scores and human perception of text-video alignment.
• Models must be evaluated at standardized resolutions (512x512 or 512x320) after generation; failing to resize videos with different aspect ratios before comparison introduces bias.
• Closed-source models were tested with default parameters on specific platforms, which may not reflect their full capability or standardized benchmarking conditions.

Evidence (verbatim from paper)

I2VGen-x1 and SVD lean towards camera movement rather than subject motion, which is why they score high on the flow-square-mean but obtain low GenVideo-RefVideo Clip scores.

Citation

@misc{fan2024aigcbench,
  title={AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI},
  author={Fan et al. (2024)},
  year={2024},
  note={arXiv:2401.01651}
}

• arXiv: 2401.01651