Vga Bench Eval

vga-bench-eval

VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation — Jiang et al. (2026) (arXiv:2604.10127, 2026)

What this evaluates

Evaluates text-to-video generation models across three dimensions: aesthetic quality, aesthetic tagging, and generation fidelity. It measures how well automated neural assessors align with human judgments and ranks models based on normalized scores across multiple visual and formal dimensions.

Datasets

• VGA-Bench — total 1700; splits: train (1300), test (400)

Metrics

• five-class accuracy (primary) — range: percent
  • Accuracy computed after discretizing the 0–10 human score scale into five levels. Calculated as the proportion of correctly predicted class labels out of total test instances.
• Top-2 tag accuracy — range: percent
  • Proportion of instances where the ground truth tag appears in the model's top two predicted tags.
• Recall@1/3/5 — range: percent
  • Measures alignment with human rankings. For each prompt, checks if the human-preferred model appears in the top K models predicted by the neural assessor, averaged across prompts.

Input / output format

Input: Video clip paired with a text prompt and/or aesthetic dimension label.

Output: Discrete class label (1–5 for aesthetic score, categorical tags for aesthetic tagging, or quality level for generation).

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)

def compute_top2_tag_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if g in p[:2]) / len(gold)

def compute_recall_k(pred_rankings, gold_rankings, k):
    hits = sum(1 for p, g in zip(pred_rankings, gold_rankings) if g in p[:k])
    return hits / len(gold_rankings)

# Final metric: average normalized scores across sub-dimensions within each main dimension

Common pitfalls

• The 0–10 human scoring scale is discretized into five levels for evaluation; treating it as a continuous regression target will yield incorrect results.
• Models are ranked by normalizing and averaging scores across all sub-dimensions within each main dimension, which can obscure performance on specific visual aspects like composition or lighting.
• Training and test sets are strictly separated by model provenance (3 models for training, 3 for testing) to prevent data leakage; mixing models across splits invalidates cross-model generalization claims.

Evidence (verbatim from paper)

for the aesthetic tag task, standard accuracy (Acc) is used as the metric; for aesthetic scoring, the 0–10 scale is discretized into five levels, and five-class accuracy is computed. Results are presented in Table 2 and Table 3.

Citation

@misc{jiang2026vgabench,
  title={VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation},
  author={Jiang et al. (2026)},
  year={2026},
  note={arXiv:2604.10127}
}

• arXiv: 2604.10127