qhjqhj00 multiui-eval
Harnessing Webpage UIs for Text-Rich Visual Understanding — Liu et al. (2024) (arXiv:2410.13824, 2024)
What this evaluates
Evaluates multimodal models' ability to understand and interact with complex webpage UIs, perform text-rich visual grounding, and generalize to OCR and document understanding tasks.
Datasets
- VisualWebBench — total ?; splits: test (-1)
- Mind2Web — total ?; splits: test (-1)
- DocVQA — total ?; splits: test (-1)
- ChartQA — total ?; splits: test (-1)
Metrics
element accuracy(primary) — range: percent- Percentage of correctly predicted UI elements or actions out of the total ground truth instances. Calculated as (correct predictions / total instances) * 100.
accuracy— range: percent- Standard exact-match or fuzzy-match accuracy for VQA and grounding tasks, depending on the specific benchmark protocol.
Input / output format
Input: High-resolution UI screenshots processed via dynamic patching, paired with text-based multimodal instructions or questions.
Output: Text responses containing answers, bounding box coordinates for grounding tasks, or action sequences for agent tasks.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = 0
for pred, gold_item in zip(predictions, gold):
if normalize(pred) == normalize(gold_item):
correct += 1
return (correct / len(gold)) * 100
Common pitfalls
- Baselines were re-implemented with identical training data to control for backbone differences, so direct comparison with original checkpoint results is invalid.
- Dynamic high-resolution image processing splits images into patches; evaluation consistency requires replicating this exact patching strategy.
- Two-stage training means final evaluation reflects both GUI-specific knowledge and general multimodal capabilities, not just web UI understanding.
Evidence (verbatim from paper)
resulting in a 7.3M-sample dataset that enables significant gains in web UI tasks (48% improvement on VisualWebBench, 19.1% boost in element accuracy on Mind2Web) and unexpected generalization to non-UI domains like document understanding, OCR, and chart interpretation, demonstrating the broad utility of web UIs for text-rich visual understanding.
Citation
@misc{liu2024multiui,
title={Harnessing Webpage UIs for Text-Rich Visual Understanding},
author={Liu et al. (2024)},
year={2024},
note={arXiv:2410.13824}
}
- arXiv: 2410.13824