Webcoderbench Eval

webcoderbench-eval

WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics — Liu et al. (2026) (arXiv:2601.02430, 2026)

What this evaluates

Evaluates LLMs' ability to generate complete web applications from real-world user requirements. It probes multi-modal understanding, code generation quality, and strict adherence to ground-truth checklists across functionality, visual design, and content dimensions.

Datasets

• WebCoderBench — total 1572; splits: test (1572)

Metrics

• checklist-based evaluation (primary) — range: [0, 1]
  • Models are scored based on how many points from the ground-truth checklists (covering functionality, visual design, and content) are correctly implemented in the generated web app artifact. Scores are aggregated across the three dimensions.

Input / output format

Input: Natural language user requirements, optionally accompanied by reference images or URLs intended as page content rather than design screenshots.

Output: Complete web application code/artifact corresponding to the provided requirement.

Scoring recipe

def score_checklist(predictions, gold_checklists):
    # predictions: generated web app code
    # gold_checklists: dict with keys 'functionality', 'visual_design', 'content'
    score = 0
    for dimension in ['functionality', 'visual_design', 'content']:
        points = gold_checklists[dimension]
        matched = count_matched_points(predictions, points)
        score += matched / len(points)
    return score / 3  # Average across dimensions

Common pitfalls

• Multi-turn requirements are pre-merged into single-turn by humans and LLMs, so the benchmark does not evaluate iterative refinement or chat-based interaction.
• Images and URLs are provided as page content references, not as screenshot-to-code design targets, which differs from typical vision-to-code benchmarks.
• Ground-truth checklists are synthesized via LLM inference and human validation, introducing potential subjective bias in the evaluation criteria.

Evidence (verbatim from paper)

Finally, to enable objective evaluation, we establish ground-truth checklists for each requirement across three dimensions: functionality, visual design, and content. We adopt three LLMs (GPT-5-Chat-2025-08-07, Gemini-2.5-pro, and Doubao-Seed-1.6) to infer ground-truth checklists for each dimension. After that, human experts merge and validate the outputs to produce the final ground-truth checklists.

Citation

@misc{liu2026webcoderbench,
  title={WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2601.02430}
}

• arXiv: 2601.02430