Openve Bench Eval

openve-bench-eval

OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing — He et al. (2025) (arXiv:2512.07826, 2025)

What this evaluates

Evaluates instruction-guided video editing models on spatially-aligned and non-spatially-aligned editing tasks, probing temporal consistency, spatial fidelity, and instruction following.

Datasets

• OpenVE-Bench — total ?; splits: test (-1)

Metrics

• overall score (primary) — range: other
  • LLM-based evaluation using Seed-1.6VL and Gemini 2.5 Pro evaluators, averaged into a single overall score. Scores appear to be on a 1–5 scale based on ablation table values.

Input / output format

Input: Original video, editing instruction text, and optionally control signals (e.g., Canny edge maps or depth maps).

Output: Edited video that applies the requested changes while preserving temporal and spatial consistency.

Scoring recipe

def compute_score(predictions, gold):
    scores_seed = run_llm_judge("Seed-1.6VL", predictions, gold)
    scores_gemini = run_llm_judge("Gemini 2.5 Pro", predictions, gold)
    return average(scores_seed + scores_gemini)

Common pitfalls

• Baseline models have strict resolution/frame limits (e.g., OmniVideo capped at 640x352/17 frames, ICVE downsampled to 384x240 to avoid OOM).
• Closed-source Runway Aleph was only evaluated on 30 samples per category due to API cost constraints, which may not represent full benchmark performance.

Evidence (verbatim from paper)

Tab. 2 and 3 present the evaluation results of all instruction-following video editing models on OpenVE-Bench. The closed-source model, Runway Aleph, achieves the best performance on both the Seed-1.6VL [34] and Gemini 2.5 Pro [10] evaluators, significantly outperforming existing open-source models. The open-source models VACE [17], OmniVideo [33], and InsViE [44] exhibit limited performance due to constraints in their model size or training data. With 5B parameters, Lucy-Edit [35] achieves moderate performance. ICVE [25], with 13B parameters, obtains respectable results, but its high-resolution editing capability is limited to a smaller number of frames. Ditto [2] scores highly on the global style metric, as its training dataset primarily consists of this edit type. Our model, OpenVE-Edit, with only 5B parameters, achieves an overall score of 2.41, surpassing all existing open-source models with a smaller parameter count.

Citation

@misc{he2025openve3m,
  title={OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2512.07826}
}

• arXiv: 2512.07826