Videoaesbench Eval

videoaesbench-eval

VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models — Li et al. (2026) (arXiv:2601.21915, 2026)

What this evaluates

Evaluates large multimodal models' ability to perceive and judge video aesthetics across visual form, style, and affectiveness dimensions. It tests performance on diverse video sources (UGC, AIGC, RGC, compression, gaming) using multiple-choice, true/false, and open-ended questions.

Datasets

• VideoAesBench — total 1804; splits: test (1804)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. For closed-ended questions (True/False, single/multiple choice), a prediction is correct if it exactly matches the ground truth option(s). For open-ended questions, scores are typically derived via LLM-judge or human evaluation against reference captions, though the exact automated protocol is not detailed in the provided text.

Input / output format

Input: A video clip paired with a question prompt. Questions are categorized as True/False, single choice (4 options), multiple choice (4 options, >1 correct), or open-ended.

Output: For True/False: "True" or "False". For single/multiple choice: the selected option letter(s) or text. For open-ended: free-text aesthetic description or suggestion.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• True/False questions must be balanced in ratio to avoid bias from unbalanced answer distributions.
• Multiple-choice questions require selecting all correct options; partial credit is not mentioned, so exact set matching is likely required.
• Open-ended questions lack predefined answer sets, making automated scoring unreliable without explicit LLM-judge prompts or human annotators.

Evidence (verbatim from paper)

The goal of LMMs is to select the most accurate answer from the set of options including distractors. This type of question can detailedly assess the capability of LMMs in understanding and distinguishing diverse aesthetics issues.

Citation

@misc{li2026videoaesbench,
  title={VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2601.21915}
}

• arXiv: 2601.21915