Graphic Design Bench Eval

graphic-design-bench-eval

Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks — Deganutti et al. (2026) (arXiv:2604.04192, 2026)

What this evaluates

Evaluates AI systems' ability to perceive, reason about, and generate professional graphic design artifacts across layout, typography, vector graphics, template semantics, and animation. It probes multi-constraint satisfaction through design-native metrics measuring spatial accuracy, perceptual quality, and semantic alignment.

Datasets

• LICA layered-composition dataset — total ?; splits: test (-1); repo https://github.com/purvanshi-lica/lica-bench

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union; measures average overlap between predicted and ground-truth bounding boxes.
• CLIP Score — range: [0, 1]
  • Cosine similarity between image and text embeddings, measuring semantic alignment.
• LPIPS — range: [0, 1]
  • Learned perceptual distance between two images using deep features; lower values indicate greater perceptual similarity.
• SSIM — range: [0, 1]
  • Structural similarity index based on luminance, contrast, and structure; higher is better.
• FID — range: [0, 1]
  • Fréchet Inception Distance; measures distributional distance between generated and reference image sets in Inception feature space.

Input / output format

Input: Varies by task: text-only (layout JSON or text metadata), image-only (rendered PNG), or both. For video tasks, native video or uniformly sampled keyframes. All evaluated with greedy decoding (temperature=0) using fixed prompt templates.

Output: Task-dependent: natural language/text responses for understanding/classification, rendered images, SVG code, Lottie JSON, or video files for generation tasks.

Scoring recipe

def score(predictions, gold):
    # Spatial accuracy (mIoU)
    ious = [intersection_over_union(pred_box, gold_box) for pred_box, gold_box in zip(predictions, gold)]
    mIoU = sum(ious) / len(ious)
    # Semantic alignment (CLIP Score)
    clip_scores = [cosine_similarity(clip_encode(pred), clip_encode(gold)) for pred, gold in zip(predictions, gold)]
    # Perceptual quality (LPIPS)
    lpips_scores = [lpips_metric(pred_img, gold_img) for pred_img, gold_img in zip(predictions, gold)]
    return {"mIoU": mIoU, "CLIP_Score": mean(clip_scores), "LPIPS": mean(lpips_scores)}

Common pitfalls

• Modality mismatch: Not all models support native video or specific input types; keyframe extraction is used as a fallback, which may skew temporal reasoning results.
• Sample size disparity: Understanding tasks use ~1,000-2,500 samples, while generation tasks are limited to ~100 due to API costs, affecting statistical power.
• Expert-defined solvability thresholds: Performance is categorized as 'Mostly Solved' (>95%), 'Partially Solved' (80-95%), or 'Unsolved' (<80%) based on design expert consultation, not standard academic baselines.

Evidence (verbatim from paper)

Throughout the paper we assign each task one of three solvability labels. Mostly Solved: best-model performance exceeds 95% (or an equivalent metric-specific threshold), with limited room for improvement at the current evaluation granularity. Partially Solved: best-model performance falls between 80–95%... mIoU (mean Intersection over Union) measures average overlap between predicted and ground-truth bounding boxes. CLIP Score computes cosine similarity between image and text embeddings. LPIPS computes learned perceptual distance between two images using deep features; lower values indicate greater perceptual similarity.

Citation

@misc{deganutti2026graphicdesignbench,
  title={Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks},
  author={Deganutti et al. (2026)},
  year={2026},
  note={arXiv:2604.04192}
}

• arXiv: 2604.04192