Univbench Eval

univbench-eval

UniVBench: Towards Unified Evaluation for Video Foundation Models — Jianhui Wei et al. (2026) (arXiv:2602.21835, 2026)

What this evaluates

Evaluates video foundation models across six core tasks (understanding, generation, editing, reconstruction) by scoring their outputs on eight cinematic and semantic dimensions, including subject consistency, action dynamics, camera movement, and lighting.

Datasets

• UniVBench — total 200; splits: test (200); repo https://github.com/JianhuiWei7/UniVBench

Metrics

• UniV-Eval (primary) — range: percent
  • Average percentage score across eight predefined cinematic and semantic dimensions: Subject, Background, Action, Camera, Color, Lighting, Video Style, and Relative Position. Computed by the UniV-Eval agentic system using standardized prompts and rubrics.

Input / output format

Input: Per task: ground-truth captions (T2V), source videos + editing instructions (TV2V), reference images + prompts (R2V), or raw videos (V2T, V2V). Videos are center-cropped and resized to model-specific resolutions while maintaining aspect ratio.

Output: Generated or edited videos for generative/editing tasks; text captions for understanding tasks. All outputs are processed by the UniV-Eval system for scoring.

Scoring recipe

def compute_univ_eval_score(predictions, gold, task_type):
    # Pass prediction through UniV-Eval agentic system
    # System applies standardized prompts and rubrics
    # Scores 8 dimensions: Subject, Background, Action, Camera,
    #    Color, Lighting, Video Style, Relative Position
    dim_scores = uni_eval_agent.score(prediction, gold, task_type)
    # Average across dimensions
    return sum(dim_scores) / len(dim_scores)

Common pitfalls

• BLEU score distortion in V2T tasks due to varying effective caption lengths across models.
• Conventional LLM-as-a-Judge approaches often lack interpretability and cover limited evaluation dimensions.
• V2V reconstruction performance heavily depends on the quality of the intermediate V2T caption, causing information transmission loss.
• Models without native task support require minimal adaptations (e.g., embedding concatenation, feature injection) that may skew results.

Evidence (verbatim from paper)

UniV-Eval implements a fine-grained dynamically adaptive evaluation strategy. All outputs undergo evaluation by our agentic system using identical prompts, rubrics, and dimension weightings, ensuring differences in scores reflect model capabilities rather than evaluation variance.

Citation

@misc{wei2026univbench,
  title={UniVBench: Towards Unified Evaluation for Video Foundation Models},
  author={Jianhui Wei et al. (2026)},
  year={2026},
  note={arXiv:2602.21835}
}

• arXiv: 2602.21835