Unipercept Bench Eval

unipercept-bench-eval

UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture — Cao et al. (2025) (arXiv:2512.21675, 2025)

What this evaluates

Evaluates multimodal large language models on perceptual-level image understanding across three domains: Image Aesthetics & Art (IAA), Image Quality Assessment (IQA), and Image Structure & Texture Assessment (ISTA). It probes both continuous visual rating (VR) and discrete visual question answering (VQA) capabilities.

Datasets

• UniPercept-Bench — total ?; splits: test (-1); repo https://github.com/thunderbolt215/UniPercept

Metrics

• Acc. (primary) — range: percent
  • Percentage of correctly predicted answers for VQA tasks, averaged across all question templates and domains.
• (SRCC+PLCC)/2 — range: [0, 1]
  • Average of Spearman Rank Correlation Coefficient (SRCC) and Pearson Linear Correlation Coefficient (PLCC) between predicted continuous scores and ground-truth ratings for VR tasks.

Input / output format

Input: An image paired with a text prompt. For VQA, the prompt is a specific question (e.g., Level Prediction, How, What, Which, Why, Yes-No). For VR, the prompt is a task-specific instruction to elicit a quantitative score.

Output: For VQA: a categorical answer string matching the ground-truth options. For VR: a continuous numerical score.

Scoring recipe

# VQA Accuracy
vqa_acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)

# Visual Rating Score
srcc = spearmanr(predictions, gold)[0]
plcc = pearsonr(predictions, gold)[0]
vr_score = (srcc + plcc) / 2

Common pitfalls

• Visual Rating tasks require generating continuous numerical scores, which often causes general MLLMs to produce hallucinated or unstable predictions compared to specialized models.
• Specialized models used for comparison are trained exclusively on in-domain datasets, so cross-domain evaluation may require retraining or careful prompt engineering to ensure fair comparison.
• VQA accuracy varies significantly by question template (e.g., Level Prediction is much harder than Yes-No), so reporting only the overall average can mask domain-specific weaknesses.

Evidence (verbatim from paper)

For VQA, all models were provided with identical prompts corresponding to each question, and their generated answers were compared against the ground-truth options. For VR, we designed task-specific prompts for models lacking a dedicated scoring interface to elicit quantitative predictions... Figure 7: Results on UniPercept-Bench.VQA and VR are evaluated by Acc. and $(SRCC+PLCC)/2$.

Citation

@misc{cao2025unipercept,
  title={UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2512.21675}
}

• arXiv: 2512.21675