# Gui Grounding Navigation Eval

> This evaluation probes a GUI agent's ability to localize UI elements via grounding and execute multi-step navigation tasks across mobile, web, and desktop platforms. It measures spatial perception, action planning consistency, and cross-platform generalization under both offline and online interaction settings. Use when the user wants to benchmark on ScreenSpot-V2, ScreenSpot-Pro, AndroidControl, AndroidWorld, ChiM-Nav, Ubu-Nav, or asks about evaluating this task. Reports success rate, Step Success Rate (SR).

- Skill: `qhjqhj00/gui-grounding-navigation-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/gui-grounding-navigation-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/gui-grounding-navigation-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/gui-grounding-navigation-eval

---


# gui-grounding-navigation-eval

> OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution — Le Zhang et al. (2026) (arXiv:2601.20380, 2026)

## What this evaluates

This evaluation probes a GUI agent's ability to localize UI elements via grounding and execute multi-step navigation tasks across mobile, web, and desktop platforms. It measures spatial perception, action planning consistency, and cross-platform generalization under both offline and online interaction settings.

## Datasets

- **ScreenSpot-V2** — total ?; splits: test (-1)
- **ScreenSpot-Pro** — total ?; splits: test (-1)
- **AndroidControl** — total ?; splits: test (-1)
- **AndroidWorld** — total ?; splits: test (-1)
- **ChiM-Nav** — total ?; splits: test (-1)
- **Ubu-Nav** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Percentage of instances where the predicted bounding box or action sequence exactly matches the ground truth. Averaged across all categories (Mobile, Desktop, Web, CAD, Dev, etc.).
- `Type Accuracy (%)` — range: percent
  - Percentage of steps where the predicted action type (e.g., Click, Type, Scroll) matches the ground truth action type, regardless of coordinates.
- `Step Success Rate (SR)` **(primary)** — range: percent
  - Percentage of tasks where the entire predicted action sequence exactly matches the ground truth sequence.
- `Coord Actions (%)` — range: percent
  - Success rate on coordinate-based actions including Click, Drag, Scroll, LeftDouble, and RightSingle.
- `Non-coord Actions (%)` — range: percent
  - Success rate on non-coordinate actions including Type, Hotkey, PressEnter, and Finish.

## Input / output format

**Input**: Screenshots of GUI interfaces paired with natural language instructions. For online benchmarks, the model receives sequential screenshots during interaction.

**Output**: Bounding box coordinates for GUI grounding tasks, or a sequence of discrete actions (e.g., Click(x,y), Type(text), Scroll, Finish) for navigation tasks.

## Scoring recipe

```python
def score_grounding(pred_bbox, gt_bbox):
    return 1 if iou(pred_bbox, gt_bbox) > 0.5 else 0

def score_navigation(pred_seq, gt_seq):
    type_acc = sum(1 for p, g in zip(pred_seq, gt_seq) if p.type == g.type) / len(gt_seq)
    step_sr = 1 if pred_seq == gt_seq else 0
    return type_acc, step_sr

# Aggregate across dataset
avg_sr = mean(score_navigation(p, g) for p, g in zip(predictions, gold))
```

## Common pitfalls

- Assuming the model uses accessibility trees or external planners; OmegaUse is evaluated as a streamlined end-to-end agent using only screenshots.
- Confusing offline trajectory planning benchmarks (AndroidControl, ChiM-Nav, Ubu-Nav) with online interactive benchmarks (AndroidWorld), which have different evaluation dynamics and success criteria.
- Mixing up coordinate-based action evaluation (Click, Drag) with non-coordinate action evaluation (Type, Hotkey) in Ubu-Nav, as they are reported separately.

## Evidence (verbatim from paper)

> Specifically, OmegaUse reaches a Type Accuracy of 87.6% and a Step Success Rate (SR) of 79.1%. These scores surpass previous leading models such as UI-Venus-Navi-72B (85.9% Type Acc. / 77.2% Step SR) and UI-TARS-72B (85.2% Type Acc. / 74.7% Step SR).

## Citation

```bibtex
@misc{zhang2026omegause,
  title={OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution},
  author={Le Zhang et al. (2026)},
  year={2026},
  note={arXiv:2601.20380}
}
```

- arXiv: 2601.20380