qhjqhj00 showui-gui-eval
ShowUI: One Vision-Language-Action Model for GUI Visual Agent — Kevin Qinghong Lin et al. (2024) (arXiv:2411.17465, 2024)
What this evaluates
Evaluates a vision-language-action model's ability to perform GUI visual grounding and task-oriented navigation across web, mobile, and online environments. It probes the model's capacity to interpret screenshots, locate interactive elements, and execute correct action sequences to complete user instructions.
Datasets
- Screenspot — total ?; splits: test (-1)
- Mind2Web — total ?; splits: test (-1)
- AITW — total ?; splits: test (-1)
- MiniWob — total ?; splits: test (-1)
Metrics
Zero-shot grounding accuracy(primary) — range: percent- Percentage of correctly grounded text or icon elements in a screenshot, evaluated separately for text and icon tracks.
Element Accuracy (Ele.Acc)— range: percent- Percentage of correctly identified interactive elements in a navigation step.
Operation F1 (Op.F1)— range: percent- F1 score of the predicted operation type (e.g., click, type) against the ground truth.
Step Success Rate (Step.SR)— range: percent- Percentage of navigation steps where the predicted action exactly matches the ground truth action.
Overall Accuracy— range: percent- Percentage of tasks completed successfully from start to finish.
Input / output format
Input: RGB screenshot of a GUI interface paired with a natural language instruction or query.
Output: For grounding: bounding box coordinates or element identifiers for the target UI component. For navigation: a sequence of discrete actions (e.g., click, type, scroll) formatted as JSON or structured tokens.
Scoring recipe
def score_grounding(pred_boxes, gt_boxes):
correct = sum(1 for p, g in zip(pred_boxes, gt_boxes) if iou(p, g) > 0.5)
return correct / len(gt_boxes) * 100
def score_navigation(pred_actions, gt_actions):
ele_acc = sum(1 for p, g in zip(pred_actions, gt_actions) if p.element == g.element) / len(gt_actions) * 100
op_f1 = f1_score([g.op for g in gt_actions], [p.op for p in pred_actions])
step_sr = sum(1 for p, g in zip(pred_actions, gt_actions) if p == g) / len(gt_actions) * 100
return ele_acc, op_f1, step_sr
Common pitfalls
- Grounding accuracy is reported separately for text and icon tracks; averaging them without tracking the split can misrepresent performance.
- Navigation metrics evaluate different granularities; Step.SR requires exact match of both element and operation, making it stricter than Ele.Acc or Op.F1.
- MiniWob evaluation strictly follows a 35-task split; deviating from this split breaks comparability with reported baselines.
Evidence (verbatim from paper)
In Tab.[4] for web navigation, we found that: (i) Instruction-tuning has a significant effect, brings 4.6% Avg. Step SR. boost over Qwen2-VL-2B. Notably, ShowUI-2B’s zero-shot yield comparable with SeeClick-9.6B which has pretrained and fine-tuning, and achieves relatively high Op. F1 (80%+).
Citation
@misc{lin2024showui,
title={ShowUI: One Vision-Language-Action Model for GUI Visual Agent},
author={Kevin Qinghong Lin et al. (2024)},
year={2024},
note={arXiv:2411.17465}
}
- arXiv: 2411.17465