# Openve Bench Eval

> Evaluates instruction-guided video editing models on spatially-aligned and non-spatially-aligned editing tasks, probing temporal consistency, spatial fidelity, and instruction following. Use when the user wants to benchmark on OpenVE-Bench, or asks about evaluating this task. Reports overall score.

- Skill: `qhjqhj00/openve-bench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/openve-bench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/openve-bench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/openve-bench-eval

---


# openve-bench-eval

> OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing — He et al. (2025) (arXiv:2512.07826, 2025)

## What this evaluates

Evaluates instruction-guided video editing models on spatially-aligned and non-spatially-aligned editing tasks, probing temporal consistency, spatial fidelity, and instruction following.

## Datasets

- **OpenVE-Bench** — total ?; splits: test (-1)

## Metrics

- `overall score` **(primary)** — range: other
  - LLM-based evaluation using Seed-1.6VL and Gemini 2.5 Pro evaluators, averaged into a single overall score. Scores appear to be on a 1–5 scale based on ablation table values.

## Input / output format

**Input**: Original video, editing instruction text, and optionally control signals (e.g., Canny edge maps or depth maps).

**Output**: Edited video that applies the requested changes while preserving temporal and spatial consistency.

## Scoring recipe

```python
def compute_score(predictions, gold):
    scores_seed = run_llm_judge("Seed-1.6VL", predictions, gold)
    scores_gemini = run_llm_judge("Gemini 2.5 Pro", predictions, gold)
    return average(scores_seed + scores_gemini)
```

## Common pitfalls

- Baseline models have strict resolution/frame limits (e.g., OmniVideo capped at 640x352/17 frames, ICVE downsampled to 384x240 to avoid OOM).
- Closed-source Runway Aleph was only evaluated on 30 samples per category due to API cost constraints, which may not represent full benchmark performance.

## Evidence (verbatim from paper)

> Tab. 2 and 3 present the evaluation results of all instruction-following video editing models on OpenVE-Bench. The closed-source model, Runway Aleph, achieves the best performance on both the Seed-1.6VL [34] and Gemini 2.5 Pro [10] evaluators, significantly outperforming existing open-source models. The open-source models VACE [17], OmniVideo [33], and InsViE [44] exhibit limited performance due to constraints in their model size or training data. With 5B parameters, Lucy-Edit [35] achieves moderate performance. ICVE [25], with 13B parameters, obtains respectable results, but its high-resolution editing capability is limited to a smaller number of frames. Ditto [2] scores highly on the global style metric, as its training dataset primarily consists of this edit type. Our model, OpenVE-Edit, with only 5B parameters, achieves an overall score of 2.41, surpassing all existing open-source models with a smaller parameter count.

## Citation

```bibtex
@misc{he2025openve3m,
  title={OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2512.07826}
}
```

- arXiv: 2512.07826