Interaction2code Eval

interaction2code-eval

Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping — Jingyu Xiao et al. (2024) (arXiv:2411.03292, 2024)

What this evaluates

Evaluates multimodal large language models' ability to generate functional interactive webpage code from interactive prototype screenshots. It specifically probes the model's capacity to capture dynamic interaction behaviors, element positioning, and visual-textual alignment, rather than just static layout reproduction.

Datasets

• Interaction2Code — total ?; splits: train (-1), test (-1); repo https://github.com/WebPAI/Interaction2Code

Metrics

• CLIP (primary) — range: [0, 1]
  • Cosine similarity between CLIP image embeddings of the rendered generated webpage and the reference webpage screenshot.
• SSIM — range: [0, 1]
  • Structural Similarity Index Measure comparing the pixel-level structural fidelity between the generated and reference webpage screenshots.
• Text — range: [0, 1]
  • Textual overlap or embedding similarity between the extracted text content of the generated code and the reference code.
• Position — range: [0, 1]
  • Positional similarity score measuring the alignment of interactive elements' coordinates between the generated and reference webpages.
• IR — range: [0, 1]
  • Interaction Rate, representing the proportion of correctly generated interactive elements relative to the ground truth.
• Usability Rate — range: [0, 1]
  • Percentage of generated interactions deemed functionally consistent with the ground truth by human annotators with front-end development experience.
• Win Rate — range: [0, 1]
  • Proportion of pairwise comparisons where the model's output is preferred over the baseline, determined by majority vote (≥3 out of 5 annotators).

Input / output format

Input: Screenshot of an interactive webpage prototype (and optionally accompanying textual descriptions).

Output: HTML/CSS/JavaScript code implementing the interactive webpage.

Scoring recipe

def compute_metrics(generated_code, ref_code, gen_img, ref_img):
    gen_img, ref_img = render_to_image(generated_code), render_to_image(ref_code)
    clip = cosine_similarity(CLIP.encode(gen_img), CLIP.encode(ref_img))
    ssim = structural_similarity(gen_img, ref_img)
    text = text_similarity(extract_text(generated_code), extract_text(ref_code))
    pos = element_position_similarity(extract_elements(generated_code), extract_elements(ref_code))
    ir = count_correct_interactions(generated_code, ref_code) / count_total_interactions(ref_code)
    return {'CLIP': clip, 'SSIM': ssim, 'Text': text, 'Position': pos, 'IR': ir}

Common pitfalls

• MLLMs significantly undergenerate interactive elements relative to static content, leading to incomplete functionality.
• Performance degrades sharply for interactions with low visual saliency (small area ratio on the screen).
• Single-modality (visual-only) inputs yield substantially poorer results than combined visual-textual inputs.

Evidence (verbatim from paper)

Gemini-1.5-flash's metrics (CLIP, SSIM, text, position, IR) of the interaction part improve from direct prompting scores (0.4737, 0.3616, 0.2809, 0.4302, 0.6738) to (0.5093, 0.3854, 0.3217, 0.4511, 0.7112) with CoT, and further to (0.5194, 0.3898, 0.3454, 0.4612, 0.7326) with Mark prompting.

Citation

@misc{xiao2024interaction2code,
  title={Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping},
  author={Jingyu Xiao et al. (2024)},
  year={2024},
  note={arXiv:2411.03292}
}

• arXiv: 2411.03292