Univa Bench Eval

univa-bench-eval

UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist — Zhengyang Liang et al. (arXiv:2511.08521, 2025)

What this evaluates

Evaluates a video agent's capabilities across generation, understanding, editing, and segmentation tasks, while probing its agentic planning and memory mechanisms. It measures how well a unified agent architecture handles long-horizon, multi-step video workflows compared to monolithic baselines.

Datasets

• UniVA-Bench — total ?; splits: test (-1)

Metrics

• CLIP Score — range: [0, 1]
  • Measures prompt following by computing the cosine similarity between the text prompt embedding and the generated video embedding using a CLIP model.
• DINO Score — range: [0, 1]
  • Measures subject consistency by computing the cosine similarity between reference image/video embeddings and generated video embeddings using a DINO model.
• MLLM Judge (primary) — range: other
  • Preference ratings assigned by a Multimodal Large Language Model acting as a judge, evaluating generated videos against a fixed set of criteria aligned with human preferences.
• Normalized QA Accuracy — range: [0, 1]
  • The proportion of correctly answered questions in long video understanding tasks, normalized according to the UniVA-Bench protocol.
• J&F-mean — range: [0, 1]
  • The harmonic mean of J-mean (Jaccard index) and F-mean (frame similarity) for video segmentation evaluation.
• Success Rate — range: [0, 1]
  • The percentage of test cases where the agent produced a structurally valid plan, defined as cases where wPED > 0.
• wPED — range: [0, 1]
  • Weighted Plan Execution Distance; a metric quantifying the quality of generated plans relative to gold plans, where lower values indicate better plan quality.

Input / output format

Input: Text prompts, reference images, source videos, and long multi-turn instructions or queries.

Output: Generated videos, QA answers, segmentation masks, and structured planning trajectories/step sequences.

Scoring recipe

def score_generation(prompt, ref, video):
    clip = compute_clip_similarity(prompt, video)
    dino = compute_dino_similarity(ref, video)
    mllm = mllm_judge.evaluate(video, prompt, criteria)
    return clip, dino, mllm

def score_understanding(pred_answers, gold_answers):
    return normalized_accuracy(pred_answers, gold_answers)

def score_segmentation(pred_masks, gt_masks):
    j = j_mean(pred_masks, gt_masks)
    f = f_mean(pred_masks, gt_masks)
    return (j + f) / 2

def score_planning(plans, gold_plans):
    success_rate = mean([1 if wPED(plan, gold) > 0 else 0 for plan, gold in zip(plans, gold_plans)])
    wped_scores = [wPED(plan, gold) for plan, gold in zip(plans, gold_plans)]
    return success_rate, wped_scores

Common pitfalls

• Automated metrics like CLIP and DINO may penalize videos that successfully follow complex narrative instructions but differ in strict frame-level similarity.
• MLLM-as-a-Judge scores require careful calibration against human preferences, though the paper claims strong alignment.
• wPED, DepCov, and ReplanQ metrics are referenced but their exact mathematical formulations are deferred to the Appendix or benchmark specification.
• Baseline model setups and hyperparameters are not detailed in the main text, requiring Appendix [9] for full reproducibility.

Evidence (verbatim from paper)

Evaluating the results using CLIP Score (prompt following), DINO Score (subject consistency), and preference ratings from an MLLM-as-a-Judge, following the UniVA-Bench specification. Performance is measured by the normalized QA accuracy score as defined in the UniVA-Bench protocol.

Citation

@misc{liang2025univa,
  title={UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist},
  author={Zhengyang Liang et al.},
  year={2025},
  note={arXiv:2511.08521}
}

• arXiv: 2511.08521