Editverse Bench Eval

editverse-bench-eval

EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning — Ju et al. (2025) (arXiv:2509.20360, 2025)

What this evaluates

Evaluates instruction-based video editing capabilities, including text alignment, temporal consistency, and editing faithfulness across diverse resolutions and orientations. It probes the model's ability to follow complex editing prompts while preserving unedited regions and maintaining high video quality.

Datasets

• EditVerseBench — total 200; splits: test (200)

Metrics

• VLM evaluation (Editing Quality) (primary) — range: other
  • Assesses editing faithfulness and instruction alignment via a vision-language model. Higher scores indicate better adherence to the editing prompt and preservation of context.
• Pick Score — range: [0, 1]
  • Frame-wise video quality metric measuring the alignment between generated frames and the text prompt using a pretrained vision-language model.
• CLIP Frame — range: [0, 1]
  • Text-image alignment score computed per frame using CLIP embeddings to measure similarity between the prompt and each video frame.
• ViCLIP Video — range: [0, 1]
  • Text-video alignment score computed over the entire video sequence using ViCLIP embeddings.
• CLIP Temporal Consistency — range: [0, 1]
  • Frame-to-frame consistency measured by CLIP similarity between adjacent frames in the generated video.
• DINO Temporal Consistency — range: [0, 1]
  • Frame-to-frame consistency measured by DINO feature similarity between adjacent frames.

Input / output format

Input: A text editing instruction paired with a source video (or image) to be edited.

Output: An edited video matching the spatial resolution and aspect ratio of the input, adhering to the provided text instruction.

Scoring recipe

def compute_metrics(predictions, gold):
    vlm_score = run_vlm_eval(predictions, gold['prompt'], gold['original_video'])
    pick_score = compute_pick_score(predictions, gold['prompt'])
    clip_frame = compute_clip_similarity(predictions, gold['prompt'], per_frame=True)
    clip_video = compute_clip_similarity(predictions, gold['prompt'], per_video=True)
    clip_temp = compute_clip_consistency(predictions, temporal=True)
    dino_temp = compute_dino_consistency(predictions, temporal=True)
    return {
        'Editing Quality': vlm_score,
        'Pick Score': pick_score,
        'CLIP Frame': clip_frame,
        'ViCLIP Video': clip_video,
        'CLIP Temporal Consistency': clip_temp,
        'DINO Temporal Consistency': dino_temp
    }

Common pitfalls

• Many existing benchmarks (e.g., V2VBench, TGVE+) only contain square videos and target training-free editing, making direct comparisons misleading without matching the instruction-based setting.
• VLM evaluation scores correlate strongly with human judgments, but commercial models may still outperform open-source ones on raw generation quality due to base model differences.
• Temporal consistency metrics (CLIP/DINO) measure adjacent frame similarity, which may not fully capture long-range motion coherence or semantic consistency across the entire video.

Evidence (verbatim from paper)

To evaluate editing performance on our proposed EditVerseBench, we use 6 metrics covering four aspects: VLM evaluation, video quality (frame-wise Pick Score), text alignment (CLIP text-image and ViCLIP text-video alignment), and temporal consistency (frame-wise CLIP and DINO consistency).

Citation

@misc{ju2025editverse,
  title={EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning},
  author={Ju et al. (2025)},
  year={2025},
  note={arXiv:2509.20360}
}

• arXiv: 2509.20360