Webcompass Eval

webcompass-eval

WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models — Lei et al. (2026) (arXiv:2604.18224, 2026)

What this evaluates

This benchmark evaluates code language models on multimodal web coding tasks, including generating, editing, and repairing web applications from text, image, or video inputs. It probes functional correctness, UI consistency, and interactive behavior using an automated agent-based pipeline and checklist-guided LLM judges.

Datasets

• WebCompass — total ?; splits: test (-1); repo https://github.com/NJU-LINK/WebCompass

Metrics

• Overall Score (primary) — range: percent
  • Arithmetic mean of all nine task-specific dimension scores (RUN, SPI, DSQ, ITG, FTI, STC, RCT, ITI, RFF).
• Runnability (RUN) — range: percent
  • Measures whether the generated web application executes without critical errors in a headless Chromium browser.
• Spec Implementation (SPI) — range: percent
  • Evaluates how accurately the generated code implements the specified functional requirements.
• Design Quality (DSQ) — range: percent
  • Assesses the visual aesthetics and layout fidelity of the generated web page.
• Instruction Targeting (ITG) — range: percent
  • Measures whether editing tasks correctly target the specified UI elements or code regions.
• Feature Integrity (FTI) — range: percent
  • Tests both preservation of existing functionality and correct implementation of new components after editing.
• Style Conformance (STC) — range: percent
  • Evaluates the fidelity of the edit outcome to the requested visual style changes.
• Root-Cause Targeting (RCT) — range: percent
  • Measures the model's ability to correctly locate the defect's root cause without introducing new errors.
• Interaction Integrity (ITI) — range: percent
  • Primarily measures regression safety, ensuring interactive layers remain functional after repair.
• Reference Fidelity (RFF) — range: percent
  • Measures visual and functional closeness to a gold reference screenshot after repair.

Input / output format

Input: Multimodal prompts (text, image, or video) specifying a web page or modification task, plus existing codebase/screenshots for editing and repair tasks.

Output: Web application code (HTML/CSS/JS, React, or Vue) or code patches/edits.

Scoring recipe

def evaluate(predictions, gold, task_type):
    # Agent runs prediction in headless Chromium via MCP
    runnability = check_execution(predictions)
    spec_impl = check_spec_compliance(predictions, gold)
    design_q = llm_judge_score(predictions, gold, 'design')
    
    if task_type in ['editing', 'repair']:
        itg = llm_judge_score(predictions, gold, 'instruction_targeting')
        fti = llm_judge_score(predictions, gold, 'feature_integrity')
        stc = llm_judge_score(predictions, gold, 'style_conformance')
        rct = llm_judge_score(predictions, gold, 'root_cause')
        iti = llm_judge_score(predictions, gold, 'interaction_integrity')
        rff = llm_judge_score(predictions, gold, 'reference_fidelity')
        
    dims = [runnability, spec_impl, design_q, itg, fti, stc, rct, iti, rff]
    return {'Overall': mean(dims), 'Dimensions': dims}

Common pitfalls

• Confusing evaluation dimensions across task types: Editing's Feature Integrity tests preservation and new-component functionality, while Repair's Interaction Integrity primarily measures regression safety.
• Assuming visual fidelity scales linearly with functional correctness; the paper explicitly notes they do not scale in lockstep, with some models outperforming others on visual axes despite comparable executability.
• Using harmonic mean for overall scores in subtask/framework breakdowns instead of the arithmetic mean used in main results, which can cause ranking reversals due to variance penalization.

Evidence (verbatim from paper)

Each task has three evaluation dimensions: Generation uses Runnability (RUN), Spec Implementation (SPI), and Design Quality (DSQ); Editing uses Instruction Targeting (ITG), Feature Integrity (FTI), and Style Conformance (STC); Repair uses Root-Cause Targeting (RCT), Interaction Integrity (ITI), and Reference Fidelity (RFF). Overall is the arithmetic mean of all nine dimension scores.

Citation

@misc{lei2026webcompass,
  title={WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models},
  author={Lei et al. (2026)},
  year={2026},
  note={arXiv:2604.18224}
}

• arXiv: 2604.18224