Android UI

Android semantic UI automation

Drive apps through the accessibility/UI hierarchy: locate real elements (button, field, link, label) and act on them, so automation works without hardcoded coordinates and survives layout changes.

When to use

• Automating any standard Android app/dialog/page.
• Finding a control ("the login button", "field named Email").
• Clicking, typing, or reading state without coordinate guessing.

Tools used

• ui_tree — dump the current window's element tree.
• ui_find — locate element(s) by text, content-desc, class, or id.
• ui_click — activate a found element.
• ui_type — set text into a found input field.
• ui_read — read visible text/state (for verification and assertions).

Procedure

1. ui_tree to understand the current surface.
2. ui_find for the target element(s); prefer stable selectors (content-desc, resource-id) over raw text.
3. If the tree is stale/missing or ambiguous → take_screenshot and fall back to android-vision; never invent coordinates.
4. ui_click / ui_type after approval of consequential actions.
5. ui_read (or screenshot) afterward to verify the result — evidence required.

Security

• Semantic targeting minimises accidental taps, but any click/type is mutative: confirm before consequential actions.
• Never enter credentials programmatically; hand off to the human when a login/password field is focused.

Related skills

android-vision · android-input · android-screen · android-apps· android-control