Creatidesign Eval

creatidesign-eval

CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design — Hui Zhang et al. (2025) (arXiv:2505.19114, 2025)

What this evaluates

This benchmark evaluates a diffusion model's ability to generate graphic designs that precisely adhere to multiple heterogeneous conditions, including primary visual subjects, secondary layout elements, and textual prompts. It probes fine-grained multi-subject preservation, semantic layout alignment, and overall compositional harmony.

Datasets

• CreatiDesign Validation Set — total 1000; splits: val (1000)

Metrics

• Avg. Score (primary) — range: percent
  • Arithmetic mean of all sub-metrics across the three evaluation categories: multi-subject preservation, semantic layout alignment, and image quality.
• IR Score — range: percent
  • Measures overall image quality by jointly capturing prompt adherence, visual appeal, and compositional harmony.
• PickScore — range: percent
  • Evaluates image-text alignment and aesthetic quality based on human preference data.
• CLIP-I — range: percent
  • Cosine similarity between CLIP image and text embeddings to measure prompt adherence.
• DINO-I — range: percent
  • Self-supervised vision transformer similarity metric for visual element preservation.
• M-DINO — range: percent
  • Multi-scale DINO similarity metric for fine-grained visual subject preservation.
• Spatial — range: percent
  • Measures accuracy of spatial positioning for layout elements and subjects.
• Sen. Acc — range: percent
  • Semantic accuracy score for textual elements in the generated layout.
• NED — range: percent
  • Normalized Edit Distance for evaluating textual element accuracy.

Input / output format

Input: Primary image(s) containing visual subjects, semantic layout annotations (text descriptions of element positions and attributes), and a global prompt.

Output: A single generated graphic design image that integrates the provided subjects, layout, and text according to the multi-condition controls.

Scoring recipe

def compute_avg_score(predictions, gold):
    scores = []
    for pred, cond in zip(predictions, gold):
        scores.append(eval_clip_i(pred, cond['prompt']))
        scores.append(eval_dino_i(pred, cond['primary_images']))
        scores.append(eval_m_dino(pred, cond['primary_images']))
        scores.append(eval_spatial(pred, cond['layout']))
        scores.append(eval_sen_acc(pred, cond['layout']))
        scores.append(eval_ned(pred, cond['layout']))
        scores.append(eval_ir_score(pred, cond['prompt']))
        scores.append(eval_pick_score(pred, cond['prompt']))
    return sum(scores) / len(scores)

Common pitfalls

• The evaluation uses a custom 1,000-sample validation set, not a standard public benchmark; results are not directly comparable to other datasets without re-evaluation.
• Sub-metrics like CLIP-I, DINO-I, and Spatial measure distinct aspects of preservation and alignment; averaging them without understanding their distinct scales may obscure specific failure modes in subject vs. layout control.
• The 400K training dataset is synthetically generated, so model performance may not generalize to real-world photographic or hand-drawn design inputs.

Evidence (verbatim from paper)

As described in Section[4.2], we evaluate model performance from two perspectives—multi-subject preservation and semantic layout alignment—to assess whether the generated designs accurately fulfill user intent. Additionally, to evaluate overall image quality, we report IR Score and PickScore, which jointly capture prompt adherence, visual appeal, and compositional harmony across the entire image.

Citation

@misc{zhang2025creatidesign,
  title={CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design},
  author={Hui Zhang et al. (2025)},
  year={2025},
  note={arXiv:2505.19114}
}

• arXiv: 2505.19114