Veu Bench Eval

veu-bench-eval

VEU-Bench: Towards Comprehensive Understanding of Video Editing — Li et al. (2025) (arXiv:2504.17828, 2025)

What this evaluates

VEU-Bench evaluates a model's ability to understand video editing by probing 19 fine-grained tasks across 10 dimensions (e.g., shot size, cut types, transitions) and three cognitive stages: recognition, reasoning, and judging. It tests whether models can identify editing components, infer their functions, and judge their effects in video content.

Datasets

• VEU-Bench — total 50000; splits: train (-1), val (-1), test (-1)

Metrics

• Scoremc — range: [1, 5]
  • Average score on the recognition task dimension, rated on a 1-5 scale based on human or LLM evaluation of answer quality.
• Scoreoe — range: [1, 5]
  • Average score on the reasoning and judging task dimensions, rated on a 1-5 scale based on human or LLM evaluation of answer quality.
• Scoreall (primary) — range: [1, 5]
  • Overall average score across all 19 tasks and 10 dimensions, rated on a 1-5 scale.

Input / output format

Input: Video frames sampled at 1 fps (maximum 64 frames) paired with a text prompt containing a question about a specific video editing element (e.g., shot size, cut type, transition function).

Output: Natural language text response answering the prompt, identifying or reasoning about the specified editing component.

Scoring recipe

scores = []
for answer in predictions:
    score = rater.evaluate(answer, gold_reference)  # Returns 1-5
    scores.append(score)
metric_value = sum(scores) / len(scores)
return metric_value

Common pitfalls

• Uniform frame sampling (1 fps) limits models' ability to perceive motion speed and dynamic changes, sometimes causing performance worse than random guessing.
• Models frequently confuse intra-frame content with inter-frame editing effects, leading to artificially high recognition scores but poor reasoning/judging performance.
• LLM evaluators tend to bias scores toward visual factual content rather than correctly judging editing intent unless pattern-matching regularization is applied.

Evidence (verbatim from paper)

Meanwhile, Oscars gains 28.3% higher performance in Scoreall compared to SOTA open-source model LLaVA-OneVision*[[19]]*. In the more challenging dimensions cut and transition, Oscars demonstrates significantly better performance than the open-source Vid-LLM state-of-the-art, with improvements of 12.2% and 10.9% on Scoremc and Scoreoe respectively, showing exceeding video editing understanding ability.

Citation

@misc{li2025veubench,
  title={VEU-Bench: Towards Comprehensive Understanding of Video Editing},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2504.17828}
}

• arXiv: 2504.17828