Videograin Eval

videograin-eval

VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing — Xiangpeng Yang et al. (arXiv:2502.17258, 2025)

What this evaluates

Evaluates a video editing model's ability to perform multi-grained edits (class, instance, and part levels) on video-text pairs. It measures semantic alignment, temporal consistency, and pixel-level fidelity of the generated edits.

Datasets

• VideoGrain Evaluation Set — total 76; splits: test (76)

Metrics

• CLIP-T (primary) — range: [0, 100]
  • Average cosine similarity between the input text prompt and all video frames.
• CLIP-F — range: [0, 100]
  • Average cosine similarity between consecutive video frames.
• Warp-Err — range: [0, 100]
  • Pixel-level difference computed by warping edited frames using optical flow from the source video (extracted via RAFT-Large).
• Q-edit — range: [0, 100]
  • Ratio of CLIP-T to Warp-Err, used as a comprehensive measure of editing quality.

Input / output format

Input: Source video frames, editing text prompt, and spatial region/mask information for target edits.

Output: Edited video frames matching the source duration and resolution.

Scoring recipe

def compute_metrics(frames, prompt, source_frames):
    clip_t = mean(cosine_similarity(prompt, frame) for frame in frames)
    clip_f = mean(cosine_similarity(frames[i], frames[i+1]) for i in range(len(frames)-1))
    flow = raft_large(source_frames)
    warped = warp(frames, flow)
    warp_err = mean_pixel_diff(frames, warped)
    q_edit = clip_t / warp_err
    return clip_t, clip_f, warp_err, q_edit

Common pitfalls

• All automatic metrics are scaled by 100 in the paper, which may cause confusion if readers expect standard [0,1] ranges.
• Warp-Err relies on RAFT-Large optical flow; errors in flow estimation for fast motion or occlusions can artificially inflate the error score.
• Q-edit is a ratio metric; division by a near-zero Warp-Err can produce outlier values, though the paper scales results for clarity.

Evidence (verbatim from paper)

Four automatic metrics are employed for evaluation: CLIP-T, CLIP-F, Warp-Err, and Q-edit, following (Wu et al., [2022]; Cong et al., [2023]). All metrics are scaled by 100 for clarity. CLIP-T calculates the average cosine similarity between the input prompt and all video frames, while CLIP-F measures the average cosine similarity between consecutive frames. Additionally, Warp-Err captures pixel-level differences by warping the edited video frames according to the optical flow of the source video, extracted using RAFT-Large (Teed & Deng, [2020]). To provide a more comprehensive measure of video editing quality, we follow (Cong et al., [2023]) and use Q-edit, defined as CLIP-T/Warp-Err.

Citation

@misc{yang2025videograin,
  title={VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing},
  author={Xiangpeng Yang et al.},
  year={2025},
  note={arXiv:2502.17258}
}

• arXiv: 2502.17258