Designbench Eval

designbench-eval

DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation — Jingyu Xiao et al. (2025) (arXiv:2506.06251, 2025)

What this evaluates

Evaluates multimodal large language models (MLLMs) on front-end web development tasks, including code generation, editing, and repair across multiple frameworks (React, Vue, Angular, HTML/CSS). It probes capabilities in visual-to-code translation, framework-specific syntax handling, code localization, and component reuse.

Datasets

• DesignBench — total ?; splits: generation (-1), edit (-1), repair (-1); repo https://github.com/WebPAI/DesignBench

Metrics

• CLIP similarity — range: [0, 1]
  • Measures semantic similarity between the generated webpage and the original design using the CLIP model.
• Compilation Success Rate (CSR) (primary) — range: [0, 1]
  • Percentage of generated code that compiles successfully without errors. Calculated as CSR = S / N, where S is the number of successfully compiled samples and N is the total number of samples.
• Code Modification Similarity (CMS) — range: [0, 1]
  • Jaccard similarity between the sets of modified line numbers in ground truth and generated code. Defined as CMS(A,B) = |A ∩ B| / |A ∪ B|, where A and B are the sets of modified line numbers.
• MLLM-as-Judge Score — range: [0, 10]
  • Score from 0 to 10 assigned by GPT-4o judging whether the model meets user requirements (edit) or resolves issues (repair). 0-3: poor, 4-6: partial, 7-8: good, 9-10: excellent.

Input / output format

Input: Multimodal prompts containing UI design images/screenshots and/or existing code context, tailored to generation, edit, or repair tasks.

Output: Front-end code output (HTML/CSS/JavaScript or framework-specific syntax like React/Vue/Angular components).

Scoring recipe

def compute_metrics(predictions, golds, images):
  csr = sum(1 for p in predictions if compile(p)) / len(predictions)
  cms = jaccard_similarity(set(gold.modified_lines), set(pred.modified_lines))
  clip_score = cosine_similarity(CLIP.encode(gold.image), CLIP.encode(pred.image))
  judge_score = gpt4o_judge(prompt, pred.code, gold.context) # returns 0-10
  return csr, cms, clip_score, judge_score

Common pitfalls

• Multimodal visual inputs can actually degrade performance for edit and repair tasks compared to code-only contexts, contrary to typical multimodal expectations.
• Framework-specific syntax handling and component reuse are major bottlenecks; models often fail on complex inputs or specific framework conventions.
• MLLM-as-judge scores require careful validation against human evaluation, as automated scoring may still exhibit bias despite high reported accuracy.

Evidence (verbatim from paper)

We evaluate the performance of the model on DesignBench from three types of metrics: Visual Metrics. CLIP is applied to measure the semantic similarity between the generated and original webpages. Code Metrics. (1) Compilation Success Rate (CSR). This metric represents the percentage of generated code that compiles successfully without errors. Assume that the total number of samples is N and the number of samples compiled successfully is S, then CSR=S/N. (2) Code Modification Similarity (CMS). We employ the Jaccard similarity to quantify the precision of code modifications on design edit and design repair tasks by comparing the sets of modified line numbers between the ground truth and generated code. Let A represent the set of line numbers modified in the ground truth code and B represent the set of line numbers modified in the generated code. The CMS is formally defined as: CMS(A,B)=|A∩B|/|A∪B|.

Citation

@misc{xiao2025designbench,
  title={DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation},
  author={Jingyu Xiao et al. (2025)},
  year={2025},
  note={arXiv:2506.06251}
}

• arXiv: 2506.06251