SAMRADDHASHRIVASTAVATECH Android vision fallback
Observe the device visually when the accessibility tree cannot be trusted, then let the higher skills act on the visual understanding.
When to use
- UI tree missing/stale (games, embedded browsers, canvas/Skia surfaces, webviews).
- The tree exists but does not match what is visibly on screen.
- Screen shows content that must be read (labels, errors, QR, media) as text.
Tools used
take_screenshot— the frame to understand.screen_info— resolution/density so visual regions map to UI tools.ui_read/ui_find— try the tree first; vision is the fallback, not the default.
Procedure
- Prefer
ui_tree/ui_find. Only if the tree is unusable, go visual. take_screenshot(task-scoped region when possible) +screen_info.- Interpret the frame (labels, element regions, states).
- Map understood regions back to
android-ui/android-inputactions, or report the reading to the user. - Delete on-screen records that are no longer needed.
Security
- Reading the screen may read private content: only capture what the task needs.
- Vision is an observation tool; it never authorises an action the gates refuse.
Related skills
android-ui · android-screen · android-input · android-control