Text Video Alignment Eval

text-video-alignment-eval

VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement — Daeun Lee et al. (2024) (arXiv:2411.15115, 2024)

What this evaluates

This evaluation protocol assesses how well text-to-video generation models align generated content with textual prompts across fine-grained attributes like object counts, colors, actions, and spatial relationships. It measures both semantic alignment and visual/motion quality to determine if refinement techniques successfully correct misalignments without degrading fidelity.

Datasets

• EvalCrafter — total ?; splits: count (-1), color (-1), action (-1), others (-1); repo https://github.com/evalcrafter/EvalCrafter/blob/master/metadata.json
• T2V-CompBench — total 300; splits: consistent_attribute_binding (100), spatial_relationship (100), numeracy (100)

Metrics

• Text-Video Alignment (primary) — range: [0, 1]
  • Average of six component scores: CLIP-Score, SD-Score, BLIP-BLEU, Detection-Score, Count-Score, and Color-Score.
• Video Quality — range: [0, 1]
  • Average of Video Quality Assessment score and Inception Score.
• Motion Quality — range: [0, 1]
  • Weighted average of Action Recognition score (VideoMAE) and Average Flow score.
• Temporal Consistency — range: [0, 1]
  • Combination of Warping Error (optical flow) and CLIP-Temp.
• T2V-CompBench Category Score — range: [0, 1]
  • Evaluated per category using ImageGrid-LLaVA for attribute binding, and GroundingDINO for spatial relationships and numeracy.

Input / output format

Input: Text prompt and an initial video generated by a text-to-video model.

Output: Refined video output after applying the localized refinement pipeline.

Scoring recipe

def compute_text_video_alignment(video, prompt):
    scores = [
        clip_score(video, prompt),
        sd_score(video, prompt),
        blip_bleu(video, prompt),
        detection_score(video, prompt),
        count_score(video, prompt),
        color_score(video, prompt)
    ]
    return sum(scores) / len(scores)

Common pitfalls

• SLD's frame-level latent merging often breaks object count and spatial consistency across frames, artificially lowering alignment scores.
• OPT2I only optimizes in text space without spatial guidance, capping alignment improvements regardless of prompt paraphrasing.
• Composite metrics like Text-Video Alignment average multiple heterogeneous evaluators; reporting only the aggregate hides category-specific failures.

Evidence (verbatim from paper)

For evaluation metrics in EvalCrafter, we mainly adopt the overall text-video alignment, video quality scores, motion quality, and temporal consistency metrics. Here, the text-video alignment score is defined as an average of CLIP-Score, SD-Score, BLIP-BLEU, Detection-Score, Count-Score, and Color-Score.

Citation

@misc{lee2024videorepair,
  title={VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement},
  author={Daeun Lee et al. (2024)},
  year={2024},
  note={arXiv:2411.15115}
}

• arXiv: 2411.15115