Vision2web Eval

vision2web-eval

Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification — He et al. (2026) (arXiv:2603.26648, 2026)

What this evaluates

Evaluates multimodal coding agents' ability to generate, deploy, and verify visual website development tasks across three hierarchical levels: static webpage generation, interactive frontend development, and full-stack website construction. It probes long-horizon planning, cross-modal reasoning, multi-page coordination, and visual/functional fidelity.

Datasets

• Vision2Web — total ?; splits: test (-1)

Metrics

• Visual Score (VS) (primary) — range: [0, 100]
  • A VLM-based judge (Gemini-3-Pro-Preview) scores the visual fidelity of the deployed website against the prototype image on a 0–100 scale.
• Functional Score (FS) (primary) — range: [0, 100]
  • A GUI agent verifier (GLM-4.6V) executes predefined test workflows on the deployed site. The score is the pass rate/functional correctness normalized to a 0–100 scale.
• Deployment Success Rate (DSR) — range: percent
  • The percentage of tasks that successfully generate a startup script, deploy on a fixed port, and run without errors within a 10-minute timeout.

Input / output format

Input: Prototype images, textual requirements, and multimedia resources provided in a working directory, along with carefully designed prompts guiding the required level of project completion.

Output: Generated project files, a startup script to run the project on a fixed port, and a successfully deployed web application.

Scoring recipe

def score_task(agent_output, gold_spec):
    # 1. Deployment check
    if not deploy(agent_output, timeout=600):
        return {"DSR": 0, "FS": 0, "VS": 0}
    # 2. Functional Score via GUI Agent Verifier
    test_results = gui_agent_verify(agent_output, gold_spec.workflows)
    FS = (sum(test_results.passed) / len(test_results)) * 100
    # 3. Visual Score via VLM Judge
    VS = vlm_judge_score(agent_output.screenshot, gold_spec.prototype)
    # 4. Deployment Success Rate
    DSR = 1.0
    return {"DSR": DSR, "FS": FS, "VS": VS}

Common pitfalls

• Deployment timeout (10 min) or errors automatically zero out all scores, which can disproportionately penalize agents that take longer to compile or run.
• Visual and Functional scores are evaluated by different specialized models (VLM judge vs GUI agent), so results are not purely end-to-end model capabilities but depend on the verifier setup.
• DSR is explicitly noted as 'for reference rather than an official metric' and should not be conflated with the primary VS/FS scores.

Evidence (verbatim from paper)

Table 3: End-to-end performance of multimodal coding agents on Vision2Web across three task levels, reporting device-wise static scores, averaged functional scores (FS) and visual scores (VS) for interactive and full-stack tasks, with Deployment Success Rate (DSR) provided for reference rather than an official metric.

Citation

@misc{he2026vision2web,
  title={Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2603.26648}
}

• arXiv: 2603.26648