Screen Analyze Skill
Understand the current screen state and locate UI elements for automation.
When to Use This Skill
- Before clicking/tapping any element
- When user asks "What's on the screen?"
- When automation actions fail (wrong location)
- To verify screen changed after an action
Tool Priority (CRITICAL)
ALWAYS use accessibility tree FIRST. Screenshots are LAST resort.
| Priority |
Tool |
Use When |
| 1 |
mobile_list_elements_on_screen |
Finding elements, getting coordinates |
| 2 |
mobile_take_screenshot |
Element not in tree, visual verify, debug |
Why This Order Matters
- Speed: Element list ~100ms vs screenshot+vision 2-5s
- Accuracy: Element bounds are pixel-perfect, visual guessing is error-prone
- Reliability: Elements have stable identifiers, visual positions vary
Element Selection
Find targets by (in order of preference):
- Identifier/resourceId - Most stable, use when available
- Text content - Exact or partial match
- Element type - Button, EditText, ImageButton, etc.
- Position pattern - Top-right for menu, bottom for nav
Click Point Calculation
Given: { x: 100, y: 200, width: 80, height: 40 }
Click: (x + width/2, y + height/2) = (140, 220)
When to Use Screenshot
- Element not in accessibility tree (custom UI, games)
- Verifying visual state (colors, images, layout)
- Debugging mismatches between tree and display
Analysis Questions
- Where am I? - App, screen, navigation position
- What can I do? - Buttons, inputs, links, gestures
- What's the state? - Loading, empty, error, success
- What's blocking? - Popups, permissions, overlays
Finding Elements
By Pattern (not position)
| Element Type |
Look For |
| Search |
magnifying glass icon, "Search" text, EditText at top |
| Submit |
"OK"/"Send"/confirm text, bottom-right |
| Close |
X icon, "Cancel", top-right |
| Back |
arrow icon top-left, BACK button |
| Menu |
hamburger icon, three dots |
| Input |
EditText, focused field with cursor |
By Type
| Looking for |
Element types |
| Buttons |
Button, ImageButton, clickable=true |
| Text input |
EditText, TextInputEditText |
| Lists |
RecyclerView, ListView items |
Output Format
Screen: [App - Screen name]
State: [Normal/Loading/Error]
Key Elements:
- [description]: (x, y)
Available Actions:
- [action]: tap (x, y)
Blockers: [None / popup / permission / etc.]
Tips
- Refresh before acting - UI changes
- Check element visibility - may be off-screen
- Account for language differences
- Wait after transitions before analyzing
1---2name: screen-analyze3description: Analyze current screen state, find UI elements, and get precise coordinates for automation. Use before performing actions or when user asks about what's on screen.4license: MIT5---67# Screen Analyze Skill89Understand the current screen state and locate UI elements for automation.1011## When to Use This Skill1213- Before clicking/tapping any element14- When user asks "What's on the screen?"15- When automation actions fail (wrong location)16- To verify screen changed after an action1718## Tool Priority (CRITICAL)1920**ALWAYS use accessibility tree FIRST. Screenshots are LAST resort.**2122| Priority | Tool | Use When |23|----------|------|----------|24| 1 | `mobile_list_elements_on_screen` | Finding elements, getting coordinates |25| 2 | `mobile_take_screenshot` | Element not in tree, visual verify, debug |2627### Why This Order Matters2829- **Speed**: Element list ~100ms vs screenshot+vision 2-5s30- **Accuracy**: Element bounds are pixel-perfect, visual guessing is error-prone31- **Reliability**: Elements have stable identifiers, visual positions vary3233### Element Selection3435Find targets by (in order of preference):361. **Identifier/resourceId** - Most stable, use when available372. **Text content** - Exact or partial match383. **Element type** - Button, EditText, ImageButton, etc.394. **Position pattern** - Top-right for menu, bottom for nav4041### Click Point Calculation4243```44Given: { x: 100, y: 200, width: 80, height: 40 }45Click: (x + width/2, y + height/2) = (140, 220)46```4748### When to Use Screenshot4950- Element not in accessibility tree (custom UI, games)51- Verifying visual state (colors, images, layout)52- Debugging mismatches between tree and display5354## Analysis Questions55561. **Where am I?** - App, screen, navigation position572. **What can I do?** - Buttons, inputs, links, gestures583. **What's the state?** - Loading, empty, error, success594. **What's blocking?** - Popups, permissions, overlays6061## Finding Elements6263### By Pattern (not position)6465| Element Type | Look For |66|--------------|----------|67| Search | magnifying glass icon, "Search" text, EditText at top |68| Submit | "OK"/"Send"/confirm text, bottom-right |69| Close | X icon, "Cancel", top-right |70| Back | arrow icon top-left, BACK button |71| Menu | hamburger icon, three dots |72| Input | EditText, focused field with cursor |7374### By Type7576| Looking for | Element types |77|-------------|---------------|78| Buttons | Button, ImageButton, clickable=true |79| Text input | EditText, TextInputEditText |80| Lists | RecyclerView, ListView items |8182## Output Format8384```85Screen: [App - Screen name]86State: [Normal/Loading/Error]8788Key Elements:89- [description]: (x, y)9091Available Actions:92- [action]: tap (x, y)9394Blockers: [None / popup / permission / etc.]95```9697## Tips9899- Refresh before acting - UI changes100- Check element visibility - may be off-screen101- Account for language differences102- Wait after transitions before analyzing
sheng1111