Overview
This skill turns the device:operator agent into a mobile-device driver. It activates when the user talks about a phone or tablet (real or simulated) and brings in the device-mobile capability — the mobile-mcp MCP server — so the agent has tools to list devices, manage apps, snapshot the UI tree, tap by accessibility-id, type, swipe, press device buttons, and rotate the screen.
The agent already carries the universal discipline (snapshot → locate → act → verify → record, run.json schema, checkpoint policy, secret redaction, output directory layout). This skill carries only what is mobile-specific: which tools exist on mobile-mcp, how to target elements on a mobile UI tree, how to handle iOS/Android prerequisites and dialogs, and which mobile-specific safety boundaries cannot be crossed.
Mental model
A phone is not a desktop. The UI is touch-first, layouts shift on orientation, every screen has an accessibility tree the OS will hand you, and the operating system itself owns half the surface (permission prompts, app-store sheets, biometric overlays, status bar). You target by semantic id when one exists; you reason about the active app, the active screen, and the active orientation as part of your state model.
Three immovable invariants on mobile:
- The active device is part of state.
list_devices runs first if the user did not pick one. If multiple are connected, you do not guess — you stop and ask. After a set_screen_size or rotation, you re-snapshot before issuing any coordinate-based action; the old coordinates are gone.
- OS overlays are not your screens. Permission prompts, biometric sheets, app-store dialogs, and the share sheet are owned by iOS / Android — not by the app you are testing. They appear unexpectedly. You stop, screenshot, and ask the user how to answer. You never auto-tap "Allow" or "Don't Allow".
- App-store, jailbreak, root, DRM, and biometric protections are off-limits. The agent's
critical block already says so; this skill restates it because mobile is where these protections live.
Tooling — what mobile-mcp gives you
The mobile-mcp MCP server exposes tool families. Use the most semantic option available before falling back.
| Family |
Use for |
| Device discovery |
list_devices, get_screen_size, set_screen_size, get_orientation, set_orientation |
| App management |
List installed apps, install_app, uninstall_app, launch_app (by bundle id / package), terminate_app |
| Screen interaction |
Take screenshot, list UI elements (accessibility tree), click, double_tap, long_press, swipe |
| Input |
type_text, press_button (home, back, volume up/down, power), open_url (deep links) |
Always start a run with list_devices if the user did not name one. Record the chosen device id, platform (ios / android), OS version, and screen size into run.json under target.
Element targeting
Mobile UIs are introspectable. Targets go through a strict preference order:
accessibility-id — the most stable selector iOS / Android expose. Always try this first.label or name — visible UI text the OS reports for the element.role — when the element is a button / textfield / link and only one matches the screen.- Exact text content — last resort among semantic options. Beware of dynamic text (counts, timestamps).
- Coordinates — fallback only when no semantic selector is present. Re-derive from a fresh snapshot every time.
If the target element is not in the tree, scroll the smallest plausible amount and re-snapshot — do not swipe blindly through the app looking for it. After three failed scroll-and-look passes, the step has failed (selector_not_found); record and stop.
After any orientation change or set_screen_size, every previous coordinate is invalid. Snapshot again before the next coordinate-based action.
Device prerequisites
The skill's device-mobile capability brings in mobile-mcp automatically. The host still needs the platform tooling.
iOS:
- Xcode and Xcode command-line tools (
xcode-select --install).
- For real devices: a paired device with developer mode enabled.
- For simulators: an iOS Simulator runtime that matches the target OS version.
Android:
- Android platform-tools on PATH (
adb resolvable in the shell).
- For real devices: USB debugging enabled; the device authorized in
adb devices.
- For emulators: a running AVD.
If a tool call fails with what looks like a missing prerequisite, stop and tell the user exactly what to install or enable. Do not retry the same tool blindly.
OS dialogs and unexpected overlays
Treat every OS-level overlay as a signal that the run cannot continue without user input:
| Overlay |
Action |
| App permission prompt (camera, contacts, notifications, location, photos, microphone, tracking) |
Stop. Screenshot. Ask how to answer. Never auto-tap Allow / Deny. |
| Biometric prompt (Face ID / Touch ID / fingerprint) |
Stop. The user must authenticate physically; you cannot. |
| App-store sign-in / payment confirmation sheet |
Stop. Confirm with the user before any tap. |
| OS update / battery / storage warning banner |
Stop. Screenshot. Let the user dismiss it. |
| Share sheet |
Treat the share sheet as a different surface; stop if it was unexpected. |
Append a failures[] row with reason: "unexpected_dialog" and last_snapshot pointing at the dialog screenshot. Set status to partial (if past a checkpoint) or failed (if not). Wait for user direction.
Mobile-specific failure modes
The agent's failures[] reason enum extends with these on mobile:
reason |
What it means |
selector_not_found |
Element with that accessibility-id / label / role was not in the tree after a fresh snapshot and up to three scroll-and-look passes |
launch_timeout |
launch_app did not reach a known ready element within 10 s |
app_not_installed |
The bundle id / package the flow needs is not present on the device; ask whether to install |
permission_denied |
An OS-level permission is blocking (push, network, location, etc.); user must grant |
unexpected_dialog |
A system / app-store / biometric sheet appeared mid-flow |
device_unavailable |
The chosen device disappeared (USB unplug, simulator quit, host suspend) |
orientation_drift |
A coordinate-based action happened after rotation without a re-snapshot — bug in the flow itself; abort and re-plan |
Be precise. Do not bucket everything into a vague reason — the next person replaying the run uses these strings to triage.
Multi-platform / regression sweeps
When the user asks to run the same flow on more than one platform (typical: iOS + Android, or two iOS versions), produce one run directory per device, each with its own run.json and checkpoints. Slug them <base-slug>--ios / <base-slug>--android so they sort together.
A diff is only useful when the same checkpoint exists on both runs. If the flow legitimately diverges (different screens, different copy), note the divergence in notes.md rather than forcing both checkpoints to match.
Safety boundary
Hard "no" — never attempt, even on request:
- Jailbreak or root the device.
- Bypass the App Store, Play Store, or sideloading protections beyond what the OS already permits for development.
- Defeat DRM, attestation (DeviceCheck, Play Integrity), or anti-tampering checks.
- Bypass biometric or screen-lock prompts.
- Auto-tap through an in-app-purchase or subscription-confirmation sheet.
- Persist credentials, OTPs, recovery codes, or session tokens to disk. They go to the device this session and stay there.
Soft "stop and confirm" — these need explicit user confirmation in this turn:
- Account deletion, password reset, sign-out from a synced account.
- Removing or reinstalling an app that holds user data.
- Sending a message, posting publicly, or initiating a paid action from the user's account.
- Changing the device language, region, or developer settings.
Checklist
References
1---2name: device-mobile-automation3description: Operate iOS and Android devices — real hardware, simulators, and emulators — via the mobile-mcp server. Covers device discovery and selection, app install / launch / terminate, accessibility-tree targeting, taps / swipes / typing / device-button presses, orientation handling, and the safety boundary around app-store / jailbreak / biometric / in-app-purchase protections. Activate when the user wants to drive a phone or tablet (iPhone, Android, iOS Simulator, Android Emulator) through a UI flow. Pairs with the device:operator agent which carries the shared snapshot → verify → record discipline; this skill carries the mobile-specific tooling and rules and pulls the device-mobile capability automatically.4license: Apache-2.05---67## Overview89This skill turns the `device:operator` agent into a mobile-device driver. It activates when the user talks about a phone or tablet (real or simulated) and brings in the `device-mobile` capability — the `mobile-mcp` MCP server — so the agent has tools to list devices, manage apps, snapshot the UI tree, tap by accessibility-id, type, swipe, press device buttons, and rotate the screen.1011The agent already carries the universal discipline (snapshot → locate → act → verify → record, `run.json` schema, checkpoint policy, secret redaction, output directory layout). This skill carries only what is mobile-specific: which tools exist on `mobile-mcp`, how to target elements on a mobile UI tree, how to handle iOS/Android prerequisites and dialogs, and which mobile-specific safety boundaries cannot be crossed.1213## Mental model1415A phone is not a desktop. The UI is touch-first, layouts shift on orientation, every screen has an accessibility tree the OS will hand you, and the operating system itself owns half the surface (permission prompts, app-store sheets, biometric overlays, status bar). You target by semantic id when one exists; you reason about the active app, the active screen, and the active orientation as part of your state model.1617Three immovable invariants on mobile:18191. The active device is part of state. `list_devices` runs first if the user did not pick one. If multiple are connected, you do not guess — you stop and ask. After a `set_screen_size` or rotation, you re-snapshot before issuing any coordinate-based action; the old coordinates are gone.202. OS overlays are not your screens. Permission prompts, biometric sheets, app-store dialogs, and the share sheet are owned by iOS / Android — not by the app you are testing. They appear unexpectedly. You stop, screenshot, and ask the user how to answer. You never auto-tap "Allow" or "Don't Allow".213. App-store, jailbreak, root, DRM, and biometric protections are off-limits. The agent's `critical` block already says so; this skill restates it because mobile is where these protections live.2223## Tooling — what mobile-mcp gives you2425The `mobile-mcp` MCP server exposes tool families. Use the most semantic option available before falling back.2627| Family | Use for |28|---|---|29| Device discovery | `list_devices`, `get_screen_size`, `set_screen_size`, `get_orientation`, `set_orientation` |30| App management | List installed apps, `install_app`, `uninstall_app`, `launch_app` (by bundle id / package), `terminate_app` |31| Screen interaction | Take screenshot, list UI elements (accessibility tree), `click`, `double_tap`, `long_press`, `swipe` |32| Input | `type_text`, `press_button` (home, back, volume up/down, power), `open_url` (deep links) |3334Always start a run with `list_devices` if the user did not name one. Record the chosen device id, platform (`ios` / `android`), OS version, and screen size into `run.json` under `target`.3536## Element targeting3738Mobile UIs are introspectable. Targets go through a strict preference order:39401. `accessibility-id` — the most stable selector iOS / Android expose. Always try this first.412. `label` or `name` — visible UI text the OS reports for the element.423. `role` — when the element is a button / textfield / link and only one matches the screen.434. Exact text content — last resort among semantic options. Beware of dynamic text (counts, timestamps).445. Coordinates — fallback only when no semantic selector is present. Re-derive from a fresh snapshot every time.4546If the target element is not in the tree, scroll the smallest plausible amount and re-snapshot — do not swipe blindly through the app looking for it. After three failed scroll-and-look passes, the step has failed (`selector_not_found`); record and stop.4748After any orientation change or `set_screen_size`, every previous coordinate is invalid. Snapshot again before the next coordinate-based action.4950## Device prerequisites5152The skill's `device-mobile` capability brings in `mobile-mcp` automatically. The host still needs the platform tooling.5354iOS:55- Xcode and Xcode command-line tools (`xcode-select --install`).56- For real devices: a paired device with developer mode enabled.57- For simulators: an iOS Simulator runtime that matches the target OS version.5859Android:60- Android platform-tools on PATH (`adb` resolvable in the shell).61- For real devices: USB debugging enabled; the device authorized in `adb devices`.62- For emulators: a running AVD.6364If a tool call fails with what looks like a missing prerequisite, stop and tell the user exactly what to install or enable. Do not retry the same tool blindly.6566## OS dialogs and unexpected overlays6768Treat every OS-level overlay as a signal that the run cannot continue without user input:6970| Overlay | Action |71|---|---|72| App permission prompt (camera, contacts, notifications, location, photos, microphone, tracking) | Stop. Screenshot. Ask how to answer. Never auto-tap Allow / Deny. |73| Biometric prompt (Face ID / Touch ID / fingerprint) | Stop. The user must authenticate physically; you cannot. |74| App-store sign-in / payment confirmation sheet | Stop. Confirm with the user before any tap. |75| OS update / battery / storage warning banner | Stop. Screenshot. Let the user dismiss it. |76| Share sheet | Treat the share sheet as a different surface; stop if it was unexpected. |7778Append a `failures[]` row with `reason: "unexpected_dialog"` and `last_snapshot` pointing at the dialog screenshot. Set status to `partial` (if past a checkpoint) or `failed` (if not). Wait for user direction.7980## Mobile-specific failure modes8182The agent's `failures[]` reason enum extends with these on mobile:8384| `reason` | What it means |85|---|---|86| `selector_not_found` | Element with that accessibility-id / label / role was not in the tree after a fresh snapshot and up to three scroll-and-look passes |87| `launch_timeout` | `launch_app` did not reach a known ready element within 10 s |88| `app_not_installed` | The bundle id / package the flow needs is not present on the device; ask whether to install |89| `permission_denied` | An OS-level permission is blocking (push, network, location, etc.); user must grant |90| `unexpected_dialog` | A system / app-store / biometric sheet appeared mid-flow |91| `device_unavailable` | The chosen device disappeared (USB unplug, simulator quit, host suspend) |92| `orientation_drift` | A coordinate-based action happened after rotation without a re-snapshot — bug in the flow itself; abort and re-plan |9394Be precise. Do not bucket everything into a vague reason — the next person replaying the run uses these strings to triage.9596## Multi-platform / regression sweeps9798When the user asks to run the same flow on more than one platform (typical: iOS + Android, or two iOS versions), produce one run directory per device, each with its own `run.json` and checkpoints. Slug them `<base-slug>--ios` / `<base-slug>--android` so they sort together.99100A diff is only useful when the same checkpoint exists on both runs. If the flow legitimately diverges (different screens, different copy), note the divergence in `notes.md` rather than forcing both checkpoints to match.101102## Safety boundary103104Hard "no" — never attempt, even on request:105106- Jailbreak or root the device.107- Bypass the App Store, Play Store, or sideloading protections beyond what the OS already permits for development.108- Defeat DRM, attestation (DeviceCheck, Play Integrity), or anti-tampering checks.109- Bypass biometric or screen-lock prompts.110- Auto-tap through an in-app-purchase or subscription-confirmation sheet.111- Persist credentials, OTPs, recovery codes, or session tokens to disk. They go to the device this session and stay there.112113Soft "stop and confirm" — these need explicit user confirmation in this turn:114115- Account deletion, password reset, sign-out from a synced account.116- Removing or reinstalling an app that holds user data.117- Sending a message, posting publicly, or initiating a paid action from the user's account.118- Changing the device language, region, or developer settings.119120## Checklist121122- [ ] `list_devices` was called and the chosen device is recorded in `run.json.target`123- [ ] Platform tooling prerequisites verified (or a clean failure with instructions for the user)124- [ ] Every action used the semantic-first targeting order (accessibility-id → label → role → text → coords)125- [ ] Snapshots taken after each rotation or screen-size change before any coordinate action126- [ ] OS / app-store / biometric / share-sheet dialogs stopped the run — none were auto-dismissed127- [ ] Failure rows use mobile-specific reasons from the enum, not generic strings128- [ ] No hard-no actions attempted (jailbreak, DRM bypass, biometric bypass, store-protection bypass)129- [ ] Soft-stop actions (account delete, paid actions, posts) had explicit user confirmation this turn130- [ ] Credentials redacted in `run.json`131- [ ] Run directory path echoed in the final response132133## References134135- [mobile-mcp on GitHub](https://github.com/mobile-next/mobile-mcp)136- [Apple — Accessibility on iOS](https://developer.apple.com/documentation/accessibility)137- [Android — UI Automator](https://developer.android.com/training/testing/other-components/ui-automator)
Muvon