Frontalk Eval

frontalk-eval

FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback — Wu et al. (2025) (arXiv:2601.04203, 2025)

What this evaluates

Probes a model's ability to generate and iteratively refine front-end code through multi-turn conversational instructions, handling both textual and visual feedback. It specifically measures functional correctness, user experience design quality, and the model's tendency to overwrite prior implementations in long-context interactions.

Datasets

• FronTalk — total ?; splits: test (-1); repo https://github.com/shirley-wu/frontalk

Metrics

• pass rate (PR) (primary) — range: percent
  • Percentage of generated code instances that successfully pass functional and visual checks verified by an automated GPT-4o-powered web agent.
• usability (UX) (primary) — range: percent
  • Score reflecting the quality of user interaction trajectories simulated by a web agent, evaluated against human-aligned automated judges based on design principles and functionality.
• forgetting rate (FR) — range: percent
  • Proportion of multi-turn tasks where the model overwrites or loses previously implemented functionality when processing new instructions for the same code component.

Input / output format

Input: A sequence of multi-turn user instructions (textual or multi-modal with UI screenshots/annotations) specifying front-end development tasks and iterative modifications.

Output: Generated front-end code (e.g., HTML/CSS/JavaScript) for each turn in the conversation.

Scoring recipe

def compute_metrics(predictions, gold):
    agent = WebAgent(model='GPT-4o')
    pr, ux, fr = [], [], []
    for pred in predictions:
        result = agent.run_code(pred.code)
        pr.append(1.0 if result.passes_functional_and_visual_checks() else 0.0)
        ux.append(agent.simulate_usability_trajectory(pred.code))
        if pred.is_multi_turn:
            fr.append(1.0 if result.overwrites_prior_functionality() else 0.0)
    return {
        'PR': sum(pr) / len(pr),
        'UX': sum(ux) / len(ux),
        'FR': sum(fr) / len(fr)
    }

Common pitfalls

• Models frequently overwrite previous code implementations instead of integrating new features, leading to high forgetting rates in multi-turn settings.
• Visual feedback interpretation is significantly harder than textual, with open-source VLMs often missing implicit functionalities or dense text annotations.
• Long-context handling degrades performance in later turns for some models, independent of the actual task difficulty.

Evidence (verbatim from paper)

For evaluation, we employ the web agent powered by GPT-4o to perform agent-based evaluation. We report pass rate (PR) and usability (UX) as main metrics and forgetting rate (FR) as a supplementary metric to measure the forgetting issue.

Citation

@misc{wu2025frontalk,
  title={FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2601.04203}
}

• arXiv: 2601.04203