Webcode2m Eval

webcode2m-eval

WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs — Gui et al. (2024) (arXiv:2404.06369, 2024)

What this evaluates

This benchmark evaluates multimodal models' ability to translate webpage design screenshots into functional HTML/CSS code. It probes visual fidelity, structural hierarchy recall, and the capacity to generate long, complex, real-world front-end code from visual inputs.

Datasets

• WebCode2M — total 2560000; splits: train (2560000), WebCode2M-Short (256), WebCode2M-Mid (256), WebCode2M-Long (256)

Metrics

• CLIP similarity — range: [0, 1]
  • Cosine similarity between the latent vectors of the reference design image and the rendered image of the generated code, both encoded by CLIP.
• Visual Score — range: [0, 1]
  • Average matching ratio between reference and candidate blocks, combined with similarity scores across four block levels: color, text, CLIP, and position.
• TreeBLEU (primary) — range: [0, 1]
  • Proportion of 1-height subtrees in the generated DOM tree that match the reference tree. Formulated as |S(t) ∩ S(ŝ)| / |S(ŝ)|, where S(·) is the set of 1-height subtrees, t is the reference tree, and ŝ is the generated tree. Terminal nodes and tag attributes are excluded.

Input / output format

Input: A screenshot/image of a webpage design, optionally accompanied by a text prompt (e.g., "write an HTML code").

Output: Raw HTML/CSS code representing the webpage design.

Scoring recipe

def compute_metrics(gold_code, pred_code, image):
    # CLIP Similarity
    img_emb = clip.encode(image)
    rendered_img = render_html_css(pred_code)
    pred_emb = clip.encode(rendered_img)
    clip_sim = cosine_similarity(img_emb, pred_emb)

    # Visual Score
    blocks_ref = extract_blocks(gold_code)
    blocks_pred = extract_blocks(pred_code)
    visual_score = avg(match_ratio(blocks_ref, blocks_pred),
                       level_sim(blocks_ref, blocks_pred, ['color','text','clip','pos']))

    # TreeBLEU
    dom_gold = parse_dom(gold_code, remove_attributes=True)
    dom_pred = parse_dom(pred_code, remove_attributes=True)
    subtrees_gold = get_1_height_subtrees(dom_gold)
    subtrees_pred = get_1_height_subtrees(dom_pred)
    treebleu = len(subtrees_gold & subtrees_pred) / len(subtrees_pred)
    return clip_sim, visual_score, treebleu

Common pitfalls

• TreeBLEU explicitly excludes terminal nodes and tag attributes (e.g., content, style) when extracting 1-height subtrees; including them will inflate scores incorrectly.
• All baselines must use a one-pass generation strategy to ensure fair comparison with specialized models fine-tuned on single-turn datasets.
• Models must handle extreme aspect ratios and variable sequence lengths without degradation, as the base architecture is specifically noted for this robustness.

Evidence (verbatim from paper)

TreeBLEU is defined as the proportion of all 1-height subtrees (see Algorithm 1) in a given tree that can be matched with that of a reference tree. Let S(·) be the set of 1-height subtrees, then it can be formulated as: TreeBLEU = |S(t) ∩ S(ŝ)| / |S(ŝ)|, where t and ŝ denote the given and reference trees, respectively.

Citation

@misc{gui2024webcode2m,
  title={WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs},
  author={Gui et al. (2024)},
  year={2024},
  note={arXiv:2404.06369}
}

• arXiv: 2404.06369