Visualwebbench Eval

visualwebbench-eval

VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding? — Liu et al. (2024) (arXiv:2404.05955, 2024)

What this evaluates

Evaluates multimodal LLMs' ability to understand web pages and ground UI elements. It probes capabilities across seven subtasks including image captioning, web question answering, OCR, element/action grounding, and action prediction.

Datasets

• VisualWebBench — total 1500; splits: test (1500)

Metrics

• Average Score (primary) — range: [0, 100]
  • Arithmetic mean of accuracy percentages across seven subtasks (Caption, WebQA, HeadOCR, OCR, Element Grounding, Action Prediction, Action Grounding), scaled to a maximum of 100.

Input / output format

Input: High-resolution webpage screenshot (1280px width) paired with a natural language prompt or question.

Output: Text response for captioning/QA/prediction tasks; selected option index for multiple-choice grounding/OCR tasks; or bounding box/point coordinates for free-form grounding tasks.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

def compute_average_score(task_accuracies):
    return sum(task_accuracies) / len(task_accuracies)

Common pitfalls

• Models limited to 448x448 input resolution struggle with 1280px wide screenshots, missing fine-grained details.
• GUI-specialized MLLMs often suffer catastrophic forgetting on general instruction-following after GUI grounding pretraining.
• Grounding evaluation uses two distinct protocols: multiple-choice selection (default) vs. free-form coordinate prediction (REC), yielding vastly different scores.

Evidence (verbatim from paper)

GPT-4V achieves an average score of only 64.6 on VisualWebBench, leaving ample room for improvement.

Citation

@misc{liu2024visualwebbench,
  title={VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2404.05955}
}

• arXiv: 2404.05955