How do I install the Fine Grained Widget Grounding skill?

Run npx skillmds@latest add dingxingdi/fine-grained-widget-grounding-2 in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Fine Grained Widget Grounding safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Fine Grained Widget Grounding?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Fine Grained Widget Grounding free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Fine Grained Widget Grounding?

dingxingdi (@dingxingdi) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Fine Grained Widget Grounding by dingxingdi

Name: Fine Grained Widget Grounding
Author: dingxingdi

Fine Grained Widget Grounding

Use this skill when the user wants tiny, crowded, or easy-to-misclick GUI targets. Trigger it for requests like “make the answer depend on a small icon,” “use repeated buttons so the agent has to pick the right one,” “test tiny checkboxes or reply icons,” or “make it fail unless it clicks the exact small widget.” This skill is for GUI tasks where precision, counting, and local disambiguation matter more than broad page understanding.

dingxingdi 81aa28e 5 files · 18.7 KB Updated Sep 17, 2026

File contents

Skill: fine-grained-widget-grounding

1. Capability Definition & Real Case

Professional Definition: The capability to localize and act on a precise GUI widget in dense, repetitive, or small-scale layouts, especially when the target belongs to a long-tail element type such as icons, toggles, radio buttons, or small action affordances. The agent must separate the true target from visually similar distractors and execute a pixel-accurate action on the right element.
Dimension Hierarchy: GUI Perception and Environment Modeling->Element Grounding->fine-grained-widget-grounding

Real Case

[Case 1]

Initial Environment: An email client shows an inbox with several messages arranged in a vertical list. Each row contains multiple compact icons, and the target action is the small reply icon associated specifically with the second email rather than the first or third.
Real Question: Click the reply icon near the second email.
Real Trajectory: Count the email rows from top to bottom, inspect the action cluster attached to the second row, distinguish the reply icon from adjacent icons such as archive or more-actions, and click that precise icon.
Real Answer: The reply composer for the second email is opened.
Why this demonstrates the capability: This case stresses fine-grained localization because multiple nearly identical icons are present and the instruction depends on ordinal position. The agent must jointly solve row counting, icon classification, and precise pointing under a dense layout. That makes the task a direct probe of high-resolution widget grounding rather than generic page comprehension.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md
Phase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md
Phase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md

---name: fine-grained-widget-grounding-2description: Use this skill when the user wants tiny, crowded, or easy-to-misclick GUI targets. Trigger it for requests like “make the answer depend on a small icon,” “use repeated buttons so the agent has to pick the right one,” “test tiny checkboxes or reply icons,” or “make it fail unless it clicks the exact small widget.” This skill is for GUI tasks where precision, counting, and local disambiguation matter more than broad page understanding.---# Skill: fine-grained-widget-grounding## 1. Capability Definition & Real Case* **Professional Definition**: The capability to localize and act on a precise GUI widget in dense, repetitive, or small-scale layouts, especially when the target belongs to a long-tail element type such as icons, toggles, radio buttons, or small action affordances. The agent must separate the true target from visually similar distractors and execute a pixel-accurate action on the right element.* **Dimension Hierarchy**: GUI Perception and Environment Modeling->Element Grounding->fine-grained-widget-grounding### Real Case**[Case 1]*** **Initial Environment**: An email client shows an inbox with several messages arranged in a vertical list. Each row contains multiple compact icons, and the target action is the small reply icon associated specifically with the second email rather than the first or third.* **Real Question**: Click the reply icon near the second email.* **Real Trajectory**: Count the email rows from top to bottom, inspect the action cluster attached to the second row, distinguish the reply icon from adjacent icons such as archive or more-actions, and click that precise icon.* **Real Answer**: The reply composer for the second email is opened.* **Why this demonstrates the capability**: This case stresses fine-grained localization because multiple nearly identical icons are present and the instruction depends on ordinal position. The agent must jointly solve row counting, icon classification, and precise pointing under a dense layout. That makes the task a direct probe of high-resolution widget grounding rather than generic page comprehension.## Pipeline Execution InstructionsTo synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially:1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md`2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md`3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`

Frequently asked questions