Vivd 10m Eval

vivd-10m-eval

VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing — Hu et al. (2024) (arXiv:2411.15260, 2024)

What this evaluates

Evaluates video editing models on local, entity-level modifications (addition, modification, deletion) by measuring background preservation, text alignment, temporal consistency, and visual quality.

Datasets

• VIVID-10M-Eval — total ?; splits: test (-1)

Metrics

• Background Preservation (BP) — range: other
  • L1 distance between the original and edited videos in non-editing regions.
• Text Alignment (TA) (primary) — range: other
  • CLIP-score of the edited region.
• Temporal Consistency (TC) — range: [-1, 1]
  • Cosine similarity between consecutive frames in the CLIP-Image feature space.
• Visual Quality (VQ) — range: [0, 1]
  • Human-annotated win rate reflecting realness and aesthetics.

Input / output format

Input: Input video frames, text caption/prompt, and a spatial mask defining the local editing region.

Output: Edited video frames matching the input resolution and frame count.

Scoring recipe

def compute_metrics(original_video, edited_video, mask, text_caption, clip_model):
    # Background Preservation (lower is better)
    bp = np.mean(np.abs(original_video - edited_video) * (1 - mask))
    # Text Alignment (higher is better)
    ta = clip_model.score(edited_region, text_caption)
    # Temporal Consistency (higher is better)
    tc = cosine_similarity(clip_model.encode(frame_t), clip_model.encode(frame_t+1))
    # Visual Quality (win rate from user study)
    vq = (wins + draws) / total_votes
    return bp, ta, tc, vq

Common pitfalls

• BP is a distance metric where lower values indicate better performance, which is counterintuitive for typical accuracy metrics.
• TC measures semantic consistency via CLIP features rather than pixel-level stability, so it may overlook visual jitter or flicker.
• User study results are reported as 'win-or-draw' rates rather than strict win rates, affecting direct comparison with other benchmarks.

Evidence (verbatim from paper)

Background Preservation (BP): the L1 distance between the original and edited videos in non-editing regions. Text Alignment (TA): the CLIP-score*[[37], [10]]* of the edited region. Temporal Consistency (TC): the cosine similarity between consecutive frames in the CLIP-Image feature space*[[37]]*.

Citation

@misc{hu2024vivid10m,
  title={VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing},
  author={Hu et al. (2024)},
  year={2024},
  note={arXiv:2411.15260}
}

• arXiv: 2411.15260