# Fine Grained Widget Grounding

> Use this skill when the user wants tiny, crowded, or easy-to-misclick GUI targets. Trigger it for requests like “make the answer depend on a small icon,” “use repeated buttons so the agent has to pick the right one,” “test tiny checkboxes or reply icons,” or “make it fail unless it clicks the exact small widget.” This skill is for GUI tasks where precision, counting, and local disambiguation matter more than broad page understanding.

- Skill: `dingxingdi/fine-grained-widget-grounding-2` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add dingxingdi/fine-grained-widget-grounding-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dingxingdi/fine-grained-widget-grounding-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: dingxingdi (https://skillmd.com/u/dingxingdi)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/dingxingdi/fine-grained-widget-grounding-2

---


# Skill: fine-grained-widget-grounding
## 1. Capability Definition & Real Case
* **Professional Definition**: The capability to localize and act on a precise GUI widget in dense, repetitive, or small-scale layouts, especially when the target belongs to a long-tail element type such as icons, toggles, radio buttons, or small action affordances. The agent must separate the true target from visually similar distractors and execute a pixel-accurate action on the right element.
* **Dimension Hierarchy**: GUI Perception and Environment Modeling->Element Grounding->fine-grained-widget-grounding
### Real Case
**[Case 1]**
* **Initial Environment**: An email client shows an inbox with several messages arranged in a vertical list. Each row contains multiple compact icons, and the target action is the small reply icon associated specifically with the second email rather than the first or third.
* **Real Question**: Click the reply icon near the second email.
* **Real Trajectory**: Count the email rows from top to bottom, inspect the action cluster attached to the second row, distinguish the reply icon from adjacent icons such as archive or more-actions, and click that precise icon.
* **Real Answer**: The reply composer for the second email is opened.
* **Why this demonstrates the capability**: This case stresses fine-grained localization because multiple nearly identical icons are present and the instruction depends on ordinal position. The agent must jointly solve row counting, icon classification, and precise pointing under a dense layout. That makes the task a direct probe of high-resolution widget grounding rather than generic page comprehension.

## Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially:

1. **Phase 1: Environment Exploration**
   Read the exploration guidelines to discover raw knowledge seeds:
   `references/EXPLORATION.md`

2. **Phase 2: Trajectory Selection**
   Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
   `references/SELECTION.md`

3. **Phase 3: Data Synthesis**
   Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
   `references/SYNTHESIS.md`