Webgen Bench Eval

webgen-bench-eval

WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning — Lu et al. (2025) (arXiv:2509.22644, 2025)

What this evaluates

Evaluates a model's ability to generate functional and visually accurate website codebases from natural language instructions. It measures both functional correctness via automated GUI-agent testing and visual fidelity via VLM-based appearance scoring.

Datasets

• WebGen-Bench — total 101; splits: test (101); repo https://github.com/mnluzimu/WebGen-Agent

Metrics

• Accuracy (primary) — range: percent
  • Percentage of instructions where the generated website passes functional tests. Calculated as (count(Yes) + 0.5 * count(Partial)) / total_instructions * 100, where 'Yes' means all 647 GUI-agent tests pass, 'Partial' means some pass, and 'No' means none pass.
• Appearance Score — range: [1, 5]
  • Average visual quality score assigned by a VLM (GPT-4o or Qwen2.5-VL-32B-Instruct) to screenshots of the rendered website. Scores range from 1 to 5.

Input / output format

Input: Natural language instruction describing the desired website functionality and layout.

Output: Complete website codebase (HTML, CSS, JavaScript files) ready for rendering.

Scoring recipe

def compute_metrics(predictions, gold_instructions):
    total = len(gold_instructions)
    yes_count = 0
    partial_count = 0
    for inst, code in zip(gold_instructions, predictions):
        test_results = run_gui_agent_tests(code, num_tests=647)
        if all(test_results):
            yes_count += 1
        elif any(test_results):
            partial_count += 1
    accuracy = (yes_count + 0.5 * partial_count) / total * 100
    appearance_scores = [vlm_score(screenshot(code)) for code in predictions]
    appearance_score = sum(appearance_scores) / len(appearance_scores)
    return accuracy, appearance_score

Common pitfalls

• Accuracy weights partial test passes at 0.5, which may undercount functionality compared to a strict pass/fail metric.
• Appearance Score is highly sensitive to the VLM used for evaluation; switching from GPT-4o to Qwen2.5-VL-32B-Instruct can shift scores by ~0.2 points.
• Start Failed cases are excluded from functional scoring but indicate environment or setup issues that should be reported separately.

Evidence (verbatim from paper)

We evaluate WebGen-Agent using WebGen-Bench, a benchmark containing 101 website-generation instructions in natural language and 647 GUI-agent test cases, covering a wide range of web applications. Following Lu et al. (2025b), we use Qwen2.5-VL-32B-Instruct in functional testing and GPT-4o in appearance evaluation. The highest Accuracy and Appearance Score are highlighted in bold.

Citation

@misc{lu2025webgenagent,
  title={WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2509.22644}
}

• arXiv: 2509.22644