Portraitcraft Eval

portraitcraft-eval

PortraitCraft: A Benchmark for Portrait Composition Understanding and Generation — Sha et al. (2026) (arXiv:2604.03611, 2026)

What this evaluates

Evaluates multimodal models on portrait composition understanding and generation. It probes the ability to predict aesthetic scores, reason about fine-grained composition attributes, answer image-grounded questions, and generate portraits that adhere to explicit spatial and compositional constraints.

Datasets

• PortraitCraft — total 50000; splits: test (-1)

Metrics

• SRCC (primary) — range: [0, 1]
  • Spearman Rank Correlation Coefficient measuring the monotonic relationship between predicted and expert composition scores.
• PLCC — range: [0, 1]
  • Pearson Linear Correlation Coefficient measuring the linear relationship between predicted and expert composition scores.
• Level Acc — range: [0, 1]
  • Accuracy of predicting the correct composition level or fine-grained attribute category.
• QA Acc — range: [0, 1]
  • Accuracy on image-grounded question answering tasks requiring composition-aware reasoning.

Input / output format

Input: Image-grounded prompts for score prediction, attribute reasoning, and QA; or text-based composition descriptions for generation.

Output: Numerical scores, categorical attribute labels, QA answers, or generated portrait images.

Scoring recipe

def score_track1(pred_scores, gold_scores, pred_levels, gold_levels, pred_qa, gold_qa):
    srcc = spearmanr(pred_scores, gold_scores)
    plcc = pearsonr(pred_scores, gold_scores)
    level_acc = mean(pred_levels == gold_levels)
    qa_acc = mean(pred_qa == gold_qa)
    return {'SRCC': srcc, 'PLCC': plcc, 'Level Acc': level_acc, 'QA Acc': qa_acc}
# Track 2 (Generation) uses qualitative evaluation only; no automated metric is specified in the protocol.

Common pitfalls

• Track 2 (Generation) relies on qualitative assessment rather than automated metrics, so quantitative comparison is not directly supported.
• Fine-grained attribute accuracy (Level Acc) shows smaller gains than global scores, indicating that attribute-level judgment remains a harder challenge than overall composition assessment.

Evidence (verbatim from paper)

Table 3 reports the baseline results on Track 1. We compare the zero-shot performance of Qwen3-VL-4B with the results obtained after partial fine-tuning on PortraitCraft. The fine-tuned model improves consistently over the zero-shot setting across all four metrics, which suggests that the proposed benchmark provides effective supervision for portrait composition understanding. The gains in SRCC and PLCC indicate better agreement with expert annotations in overall composition assessment, while the improvement in QA accuracy suggests stronger image-grounded reasoning ability after training on PortraitCraft.

Citation

@misc{sha2026portraitcraft,
  title={PortraitCraft: A Benchmark for Portrait Composition Understanding and Generation},
  author={Sha et al. (2026)},
  year={2026},
  note={arXiv:2604.03611}
}

• arXiv: 2604.03611