Gui Grounding Navigation Eval

gui-grounding-navigation-eval

OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution — Le Zhang et al. (2026) (arXiv:2601.20380, 2026)

What this evaluates

This evaluation probes a GUI agent's ability to localize UI elements via grounding and execute multi-step navigation tasks across mobile, web, and desktop platforms. It measures spatial perception, action planning consistency, and cross-platform generalization under both offline and online interaction settings.

Datasets

• ScreenSpot-V2 — total ?; splits: test (-1)
• ScreenSpot-Pro — total ?; splits: test (-1)
• AndroidControl — total ?; splits: test (-1)
• AndroidWorld — total ?; splits: test (-1)
• ChiM-Nav — total ?; splits: test (-1)
• Ubu-Nav — total ?; splits: test (-1)

Metrics

• success rate (primary) — range: percent
  • Percentage of instances where the predicted bounding box or action sequence exactly matches the ground truth. Averaged across all categories (Mobile, Desktop, Web, CAD, Dev, etc.).
• Type Accuracy (%) — range: percent
  • Percentage of steps where the predicted action type (e.g., Click, Type, Scroll) matches the ground truth action type, regardless of coordinates.
• Step Success Rate (SR) (primary) — range: percent
  • Percentage of tasks where the entire predicted action sequence exactly matches the ground truth sequence.
• Coord Actions (%) — range: percent
  • Success rate on coordinate-based actions including Click, Drag, Scroll, LeftDouble, and RightSingle.
• Non-coord Actions (%) — range: percent
  • Success rate on non-coordinate actions including Type, Hotkey, PressEnter, and Finish.

Input / output format

Input: Screenshots of GUI interfaces paired with natural language instructions. For online benchmarks, the model receives sequential screenshots during interaction.

Output: Bounding box coordinates for GUI grounding tasks, or a sequence of discrete actions (e.g., Click(x,y), Type(text), Scroll, Finish) for navigation tasks.

Scoring recipe

def score_grounding(pred_bbox, gt_bbox):
    return 1 if iou(pred_bbox, gt_bbox) > 0.5 else 0

def score_navigation(pred_seq, gt_seq):
    type_acc = sum(1 for p, g in zip(pred_seq, gt_seq) if p.type == g.type) / len(gt_seq)
    step_sr = 1 if pred_seq == gt_seq else 0
    return type_acc, step_sr

# Aggregate across dataset
avg_sr = mean(score_navigation(p, g) for p, g in zip(predictions, gold))

Common pitfalls

• Assuming the model uses accessibility trees or external planners; OmegaUse is evaluated as a streamlined end-to-end agent using only screenshots.
• Confusing offline trajectory planning benchmarks (AndroidControl, ChiM-Nav, Ubu-Nav) with online interactive benchmarks (AndroidWorld), which have different evaluation dynamics and success criteria.
• Mixing up coordinate-based action evaluation (Click, Drag) with non-coordinate action evaluation (Type, Hotkey) in Ubu-Nav, as they are reported separately.

Evidence (verbatim from paper)

Specifically, OmegaUse reaches a Type Accuracy of 87.6% and a Step Success Rate (SR) of 79.1%. These scores surpass previous leading models such as UI-Venus-Navi-72B (85.9% Type Acc. / 77.2% Step SR) and UI-TARS-72B (85.2% Type Acc. / 74.7% Step SR).

Citation

@misc{zhang2026omegause,
  title={OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution},
  author={Le Zhang et al. (2026)},
  year={2026},
  note={arXiv:2601.20380}
}

• arXiv: 2601.20380