Implicit Instruction Grounding

Skill: implicit-instruction-grounding

1. Capability Definition & Real Case

• Professional Definition: The capability to ground a user instruction to the correct GUI element when the instruction does not directly copy visible text from the interface, but instead refers to functional intent, spatial relation, or contextual semantics. The agent must infer the target control by combining screen understanding, nearby cues, and task intent rather than relying on literal lexical overlap.
• Dimension Hierarchy: GUI Perception and Environment Modeling->Element Grounding->implicit-instruction-grounding

Real Case

[Case 1]

• Initial Environment: A mobile map application is open on a permission dialog asking whether the app may access location data. The dialog contains multiple controls, including positive and negative actions, and the relevant negative action is visually small relative to the rest of the screen.
• Real Question: Reject location permission request.
• Real Trajectory: Read the permission sheet, identify that the user intent is to deny rather than approve access, map “reject” to the negative action even though the visible label is phrased differently, and tap the denial control.
• Real Answer: The location permission request is denied.
• Why this demonstrates the capability: This case is not solved by simple string matching because the user says “reject,” while the screen presents app-specific permission wording. The agent must understand the semantics of the dialog, distinguish approval from refusal, and resolve the correct action from contextual meaning rather than copied text. That combination of intent interpretation and control selection is exactly the essence of implicit grounding.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

1. Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md

2. Phase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md

3. Phase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md