Iphone Use

iphone-use — drive a real iPhone

Control a physical iPhone through the iphone-use daemon: see the screen (/agent/screenshot), act on it (/agent/input), repeat. The default backend is Direct: WebDriverAgent performs input on the phone and the device-side screen service provides pixels. It does not need iPhone Mirroring, Screen Recording, Accessibility, or the Mac cursor. The old Mirroring path is an explicit compatibility backend only.

Prerequisites

A Mac on your network running the daemon, with WDA and its 8100/9100 relays configured for the intended iPhone. Direct control needs the phone unlocked, trusted for development, and awake. Setup blockers are reported by status.

WDA itself has no authentication. Daemon bearer auth protects /agent/*, but does not protect the phone's own 8100/9100 listeners from another routable host. Use Direct only on a trusted, isolated network; when practical, turn off iPhone Wi-Fi and keep the supported Mac relays on USB loopback.

HOST="${PHONE_REMOTE_URL:-http://127.0.0.1:44321}"
AUTH="Authorization: Bearer $PHONE_REMOTE_TOKEN"   # daemon password or PHONE_REMOTE_AGENT_TOKEN
MUTATION="X-Phone-Control: 1"                      # required on every state-changing POST

Always probe first — if this fails, stop and report (don't retry blindly; 5 consecutive auth failures lock you out for 30s):

curl -s -H "$AUTH" "$HOST/agent/status"
# {"ok":true,"backend":"direct","wda":true,"wda_actionable":true,
#  "wda_locked":false,"drivable":true,"device_state":"ready",
#  "screen_state":"waiting","released":false,"reconnecting":false,"hint":"",
#  "setup_blocked_on":"", ...}

Status checks do not take control. For initialization, status/health checks, or when no unfinished user-requested task needs phone access, report the state and stop. Do not reconnect, take a hold, or poll screenshots/elements to keep the phone ready. Idle release is intentional.

Gate phone operations and screen/UI reads on drivable:true. The current user-requested task must need that access. phone_target is a legacy Mirroring-window field and is not a Direct readiness signal.

• device_state:"ready" + drivable:true → proceed with the requested task.

• device_state:"locked" → ask the operator to unlock the iPhone and keep it awake only if the current task needs phone access.

• device_state:"released" or released:true → expected idle state; leave it released unless the current task needs phone access. If that task cannot proceed, check recovery_owner:"daemon" and hint / setup_blocked_on. Once blockers are resolved and no release/reconnect is in progress, restart Direct/WDA once. This starts on-device automation, occupies the phone, and may require the operator to unlock it:

  curl -s -H "$AUTH" -H "$MUTATION" -H 'Content-Type: application/json' \
    -X POST "$HOST/agent/mode" -d '{"mode":"agent"}'
  

  With the bundled MCP server, call phone_reconnect instead under the same conditions. Then poll status until drivable:true; do not retry either path in a tight loop. Neither path accepts a transient UDID.

• device_state:"releasing" or releasing:true → do not reconnect or take a hold during release. Wait for it to finish only if the current task still needs phone access, then reassess status and the recovery conditions above.

• reconnecting:true → first inspect setup_blocked_on. If it is non-empty, follow the concrete hint; do not blindly wait or send another reconnect. Otherwise report setup_phase / setup_message, then wait and poll. A first build after an Xcode update can take several minutes. Never send input until drivable:true.

• device_state:"blocked" or "offline" → read hint and setup_blocked_on (warp|proxy|usb|trust|ddi|account). For a current task that needs phone access, resolve the blocker first; offline recovery follows the same task, ownership, and lifecycle conditions as released recovery.

• device_state:"degraded" → not a blocker; do not go looking for setup_blocked_on, it is empty by definition here. WDA answers, but the last read or action did not complete — usually a /source read that timed out on a heavy page, or a stalled app. Wait the retry_after_secs from the failing response (3s) and read again. The next health probe decides whether this clears back to ready or drops to offline; do not restart the service for it.

• Never switch to mode=mirror as automatic recovery. Mirror is an explicit operator-selected compatibility mode.

The API

If a stale caller forgets the mutation header, the 403 response names required_header:"X-Phone-Control: 1" and includes a retry hint. Correct the request once; do not repeat the unactionable POST.

Give your own client at least 40s on /agent/elements. The daemon keeps rebuilding a failed source read until its own 35s deadline before answering wda_source_failed — so a shorter client timeout (curl -m 25) cuts the connection first and hands you an empty body, not JSON. That empty body is your timeout, not a daemon fault and not an empty screen: re-read with a longer one before concluding anything about the phone.

Actions — coordinates are normalized [0,1] over the phone screen (0,0 top-left, 1,1 bottom-right), so they're resolution-independent:

curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"tap","x":0.5,"y":0.3}'
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"tap","label":"新备忘录"}'  # exact label must have one match; ambiguity sends nothing
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"tap","element":3,"snapshot":"<same elements response>"}'
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"scroll","x":0.5,"y":0.5,"dx":0,"dy":60}'
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"text","text":"Health"}'
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"key","name":"return"}'  # return|enter|send|go|search all fire the Return key — use this to submit a chat/search; coordinate-tapping a 3rd-party keyboard's 发送/前往 key ACKs but does NOT send
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"shortcut","name":"home"}'      # home|spotlight
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"longpress","x":0.4,"y":0.6,"duration_ms":700}'
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"keyboard"}'                     # dismiss the on-screen keyboard
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"set_value","element":5,"snapshot":"…","value":"你好"}'  # write a field directly (clear-then-type; "" clears); no focus tap, no keyboard dance
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"scroll","element":7,"snapshot":"…","dy":120}'          # scroll INSIDE that element's rect — never strays into a neighboring scroll view
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"alert","button":"不是 li guo?"}'  # press a system-alert button by exact name (UIAlertController; use this, NOT an element tap — alert buttons ACK a coordinate/element tap without acting)
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"alert","action":"dismiss"}'      # or the default accept/dismiss button
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"perform","element":9,"snapshot":"…","action":"increment"}'  # named affordance on that element: increment|decrement (wheel/stepper/slider), adjust (+"value"), toggle, menu (long-press menu), double_tap, two_finger_tap, scroll_to_visible, pinch, rotate, force_press

Halve the round trips: POST /agent/input?return=delta makes an applied action settle briefly and return the post-action element change in the same response — {"ok":true,"snapshot":…,"baseline":…,"delta":{…}} when the baseline (the since query param, or the action's own snapshot field) is still server-cached, else full elements. That replaces the separate act-then-GET /agent/elements verify pair for routine steps.

Read the settle block before you trust the delta. The action result and the observation are separate facts: ok:true means the action applied, and nothing in the observation can take that back — a slow or failed read comes back in settle.reason beside ok:true, never as an unknown outcome you have to escalate.

Two more flags matter: sparse:true means the tree held nothing to act on (empty, or containers only) — two identical bare reads are evidence you cannot see the screen, not that it settled, so never read sparse + unchanged as "nothing happened". stale:true means the tree in the response is the LAST good read, not a fresh one — either the refreshing read broke, or the budget cut it off mid-sample. And settled:true only ever claims the tree stopped changing — it is never a claim that what you wanted actually happened; that judgment stays yours.

settle_ms, budget_ms and waited_ms all describe the TREE read only; the alert probe is separate and hard-capped at 1.5s. Tune with settle_ms (default 15000, max 20000): drop it to 0 for a step you are going to verify by screenshot anyway. How long a read takes varies with the device and the screen — on one phone with a large tree it was measured at 6–7s — so the default leaves room for two reads rather than assuming a fast one. It is a ceiling, not a wait: reading stops as soon as two reads match. If you set a budget shorter than a single read on that screen, expect captures: 0; budget_ms in the answer tells you what it had to work with.

A failed wait_for tells you whether anyone could see the screen. Its observation block distinguishes three things you must not conflate: read:false (no tree was ever obtained — this proves nothing, least of all that an element is absent), read:true, stale:true (the screen was read, then the read path failed; you are looking at the last valid observation, not the current screen), and plain read:true (the screen was read and the condition genuinely was not met). sparse:true means the tree was empty or container-only; every absent locator it would have "satisfied" is listed in absent_unproven and the step deliberately does not pass. Never conclude an element is gone from a tree that had nothing in it — take a screenshot instead.

app_changed means your tap landed in the wrong app. Any delta response (?return=delta or GET /agent/elements?since=) grows an app_changed:{from,to} block when the frontmost app is not the one the baseline was taken in. The usual cause is a banner notification dropping in from the top and swallowing the tap, which opens its app — the delta then describes a screen you never asked for. Stop and re-orient when you see it: do not keep issuing steps against the old plan, and never treat the new screen's elements as the ones you were aiming at. Recover by sending {"type":"home"} and re-entering the intended app. Its absence is not a guarantee — a tree with no Application row reports nothing rather than guessing.

After typing into a web form the keyboard covers the page's own submit/next buttons — send {"type":"keyboard"} to dismiss it before tapping them. shortcut:"switcher" is unsupported in Direct/WDA: iOS does not expose an App Switcher action that WDA can synthesize. Do not send it and claim success.

{"type":"back"} is a left-edge-swipe gesture, not a universal Back button: on a screen with no in-app back target it can carry past the edge and switch apps (e.g. drop you onto the Home screen or another app). Prefer resolving and tapping the on-screen back control ({"type":"tap","label":"…"} / an element tap) when one exists; use back only when you know the current screen has an edge-swipe-back. For unattended runs, turn on a Focus mode first — a banner notification dropping from the top will otherwise intercept a tap and open the notifying app (hardware-seen: a chat banner hijacked a tap and opened WeChat).

MCP alternative: the repo ships iphone-use-mcp (crates/mcp) with the day-to-day safe subset: status, capabilities, reconnect, screenshot, elements, coordinate and snapshot-bound element taps, strict unique-label taps, scroll, text, named keys, Home/Spotlight, and phone_run_steps for one guarded multi-step call. phone_capabilities answers two separate questions without touching the phone: what this build supports for the configured backend, and whether the device can be driven right now (blocked_by names locked / released / reconnecting / human_handoff / owned_by_other / offline; ok: null means no evidence either way). It wakes nothing and takes no owner lease, so it is safe before deciding what to attempt. Its scope is UI control and observation only. For those seven act tools and phone_capabilities, the JSON comes back as MCP structuredContent and the text block is only a preview (trimmed at 8 KiB), so parse the structured field. phone_run_steps puts its complete batch result in both, so either is safe to parse. Every other tool keeps its original shape: phone_elements and the phone_flow_* tools return complete JSON as text (including phone_flow_run's execution result, passed or failed), and phone_screenshot returns an image. Errors raised before a call reaches the phone are explanatory text instead. The rule is simply: read structuredContent when it is there, otherwise read content according to the tool, and call phone_status / phone_capabilities when you need a machine-readable state. When a tool cannot confirm what happened it answers outcome: "unknown" with retry_safe: false — the request may have reached the phone, so check phone_elements or phone_screenshot rather than resending. Decide by the explicit retry_safe boolean, not by outcome: a local refusal and a daemon-reported not_sent are both valid evidence when they carry retry_safe: true, and not_sent on one step never means a whole batch can be replayed.

Every single-step act tool takes an optional observe. With observe: true the daemon watches the screen settle and returns settle, snapshot and delta alongside the result, so you do not need a separate phone_elements round trip to see what an action produced. It is off by default because that wait is latency the action does not otherwise pay. Read settle.reason: stable means the screen stopped changing, budget_exhausted means the observation window ran out and says NOTHING against the action itself, and observation_failed means the read broke. stale: true marks a tree from the previous successful read rather than the current screen; sparse: true marks an empty or container-only tree, which is never reported stable. Inside a sequence, tap_locator can act on the same strict label/identifier/kind/value/focus/enabled/visible locator used by wait_for; zero or multiple matches send no tap. Maintenance and less common HTTP actions such as drag, app install/uninstall, and target configuration are not exposed as native MCP tools. The installed MCP binary also runs reviewed, versioned JSON without a model: iphone-use-mcp flow validate <file> is offline, and iphone-use-mcp flow run <file> --input key=value requires Direct plus drivable:true before submitting the same guarded batch exactly once. The browser's 流程 panel can record this v1 JSON without hand-writing it: only acknowledged actions are kept, semantic labels are preferred, coordinate gestures are marked fragile, and typed text becomes a named runtime parameter without retaining the literal recorded value. Parameter values live only in the current browser page or command invocation. When the post-action element tree exposes a new unique identifier or foreground application, the recorder adds a reviewed wait_for checkpoint instead of relying on a fixed delay. It never copies arbitrary screen labels or values into an automatic checkpoint because those may contain private content. A recording that could not persist an action is an incomplete draft and cannot run from the browser. A parameterized recording can be downloaded immediately, but it cannot run until every required value is filled in.

The registry contract: check it, feed it, fix it

The official flow registry (leeguooooo/iphone-use-flows) holds reviewed, deterministic per-app scripts that replay a whole task in one call with no model and no screenshots. Every phone task you do has three registry obligations. They are not optional politeness; they are how the next run of the same task costs one call instead of thirty.

MCP="$HOME/Applications/iPhoneUse.app/Contents/MacOS/iphone-use-mcp"
"$MCP" flow update                     # once per machine; mirrors the registry
"$MCP" flow list [--category health]   # id · risk · verified · inputs · name
"$MCP" flow info health/export-all     # metadata + step templates
PHONE_REMOTE_TOKEN="$PHONE_REMOTE_TOKEN" "$MCP" flow run system/spotlight-search --input query=Health
"$MCP" flow publish my.json --as health/export-all-zh-cn --alias 健康 --note "iPhone 17 Pro Max, iOS 26"
"$MCP" flow report health/export-all --result @run.json --note "profile button is now '资料'"

MCP: phone_flow_list / info / run / update / publish / report.

1. Before acting — check. Call phone_flow_list (or flow list) before driving any app step by step. phone_elements already tells you: its registry block lists the installed flows for the app on screen and says when the store is empty (phone_flow_update once). A matching flow beats exploration every time. Read verified and risk: verified: no means nobody has proved the file on a real phone yet — run it, then take one checkpoint screenshot. risk: side_effect (send/publish/pay/delete) refuses to run without --confirm / confirm=true; confirm only after the user approved the exact target and inputs.

2. After succeeding — feed. When phone_run_steps completes 3+ steps, its registry.hint reminds you: that sequence is a flow waiting to be saved. Write it as v1 JSON (typed text → named input, page transitions guarded by wait_for, app / category / risk / locale filled in, verified_on with device, iOS, date), flow validate it, run it once from the file, then tell the user you would like to publish it and, with their OK, call phone_flow_publish(confirm=true) / flow publish. It forks if needed, adds app.json for a new app (aliases = the app's foreground label in each language, e.g. Health, 健康, so phone_elements can surface it later), rebuilds index.json, and opens the PR. Unverified files open as draft PRs. Publishing is the default outcome of a successful multi-step task, not an extra; skip it only for one-off or private tasks.

3. On failure — fix. A flow that stops (failed_step, element_not_found, missing_present) is a registry bug until proven otherwise. Read phone_elements to see where the phone stopped. If the flow is wrong (label changed, app updated, locale mismatch), tell the user and, with their OK, call phone_flow_report(id, note, confirm=true) / flow report: the last failure of that id is already captured (failed step, redacted daemon result, daemon version; screen labels and typed text are stripped). If you can also fix the locator, publish the corrected file with a bumped verified_on and mention the issue. Do not report a phone that was merely locked, offline, or on the wrong app. Never replay a failed flow blindly — the first tap may already have acted.

Compat is computed for you. Every flow in flow list / phone_flow_list / the registry block carries compat, derived from the flow's verified_on app version (the iOS version for Apple system apps) against what this phone has installed (flow apps shows the inventory):

Flow labels are locale-specific (locale): an en flow fails closed on a Chinese phone with zero matches and no tap. Pick the variant that matches the device language or record one (health/export-all-zh-cn). Keep your own flows next to the official ones with flow add <file> --as <app>/<name>; they survive flow update. Only the official source exists — there is no sources add.

The loop: see → act → verify

1. See: when drivable:true, use GET /agent/elements first — it's text (10× cheaper than vision) and carries exact labels. Fall back to screenshot when you need pixels (images, maps, unlabeled UI).
2. Act: when two or more consecutive actions are already understood, safe, and verifiable, send the longest stable segment as one phone_run_steps call with page transitions guarded by wait_for. Use one atomic action only while exploring an unknown screen, waiting for human confirmation, or isolating a failed checkpoint. Prefer phone_tap_element(element,snapshot) after choosing by identifier/kind/label/state. phone_tap_label is safe only for an exact unique label; zero or multiple matches send nothing. Use raw coordinates only when the control has no semantic target. For stateful controls (Switch / Slider / Stepper / PickerWheel), prefer perform over any tap — see the Switches & sliders rule below; when a row advertises actions (affordances flag on), use the listed action instead of guessing a gesture.
3. Verify: elements (or screenshot) again → confirm the expected change before the next step. Treat a non-2xx read or an empty tree with error as a failed checkpoint, even if an immediately preceding status said drivable:true; current daemons revoke cached actionability on either read path failure.

Operational rules. Hardware evidence is called out only where it exists; a documented or unit-tested action is not automatically a current-device proof:

• System alerts (permission prompts, "以 X 的身份设置?", sign-in confirmations): they show up as the alert:{text,buttons} block on /agent/elements, and a coordinate or element tap on an alert button is acknowledged but often does nothing (same false-success class as switches). Dismiss/answer them with {"type":"alert","button":"<exact button text>"} or {"type":"alert","action":"accept"|"dismiss"}.
• One session per phone. Name yourself on every state-changing request with X-Phone-Owner: <session-name> (the MCP server does this for you, from PHONE_REMOTE_OWNER or mcp-<pid>). Status then shows owner and owner_lease_remaining_secs. If owner is someone else, do not drive the phone: control calls answer 409 phone_owned with the owner's name — wait, ask that session to POST /agent/owner {"release":true} when it is done, and never send X-Phone-Owner-Takeover: 1 unless the user confirms the other session is abandoned. Release your own lease when your task ends.
• Human-in-the-loop pauses within a current user-requested phone task (waiting for the operator to type a password, approve a prompt, or fetch a code): use a bounded HTTP POST /agent/hold {"secs":600} before the pause if the task needs to retain the phone. Clear it with {"secs":0} when the pause ends or the task completes/is cancelled. Status reports hold_remaining_secs. A hold prevents idle release; it does not start WDA or prove readiness. Do not use or renew it for initialization, health checks, or to keep the phone ready without a task. There is no bundled MCP hold tool. A 503 device_release_in_progress (with Retry-After) means release already started. Only if the current task still needs phone access, wait for release to finish and reassess status; reconnect once under the recovery conditions above, and take another bounded hold only if the task still needs that pause.
• Switches & sliders (hardware-verified, iOS 27): a coordinate tap on a Switch is acknowledged but does not flip it — a silent false success. The only reliable path is {"type":"perform","element":N,"snapshot":"…", "action":"toggle"}, which works on both the labeled full-row switch and the bare paired UISwitch. Sliders: perform increment/decrement steps ~10%, adjust takes a normalized "value" in 0..1. Verify the new value from a fresh elements read (or the ?return=delta response) — never trust the ACK alone for state changes.
• Scroll: positive dy reveals content farther down; negative dy reveals content above. Positive dx reveals content to the right. A scroll is an atomic WDA swipe, not a stream of wheel events.
• Text input — focus and verify a field first, then {"type":"text"}. Direct/WDA sends Unicode on-device, so ASCII and CJK land without touching the Mac clipboard or keyboard.
• Named keys — the Direct implementation supports return/enter, escape, space, tab, delete/backspace, and the four arrows. Unsupported names must be treated as errors, not as successful no-ops. Re-verify these on the target iOS/WDA combination before relying on them in a destructive workflow.
• Shortcuts — Direct supports home and spotlight. switcher is unsupported. Use a supported app-launch action instead of inventing a gesture.
• WDA and iPhone Mirroring are mutually exclusive (A/B-tested on hardware): the on-phone XCUITest runner monopolizes the device's remote session, so while Direct is active any Mirroring window may show an interrupted state. That is expected. Do not try to repair or open Mirroring. Reconnect Direct only when the current task needs it and the recovery conditions above are met, using phone_reconnect or POST /agent/mode {"mode":"agent"}; it needs the phone unlocked. The target is canonical: change PHONE_REMOTE_UDID, rerun setup, and restart the daemon to switch devices. Never pass a one-off UDID during recovery.
• mode=agent stuck / wda stays false → read status.setup_blocked_on (warp|proxy|usb|trust|ddi|account). The #1 blocker is warp: Cloudflare WARP (or any VPN) wedges the CoreDevice tunnel xcodebuild needs when its effective Split Tunnel exclusions omit fe80::/10 or the device RSD ULA range fd00::/8. If WARP is only needed for selected destinations, prefer Traffic only mode with Split Tunnels Include limited to those destination IPs/CIDRs. This avoids the Local proxy mode request timeout that can break long Git uploads. Local proxy mode remains route-safe for short explicit HTTP(S) traffic. If full-tunnel WARP is still required, add both IPv6 exclusions to the Zero Trust device profile. warp-cli disconnect is only the temporary alternative. Run setup-wda.sh doctor to distinguish the two states. KeepAlive retains the last concrete blocker while its next preflight pass is checking, so an empty value means the known prerequisite checks passed. proxy means an enabled macOS HTTP/HTTPS/SOCKS entry is malformed or points at a loopback port with no listener; start that proxy app or disable only the stale entry. trust = a one-time "trust the Apple Development cert" tap on the phone.
• Reconnect only for a current task that needs phone access. Status/health checks and initialization leave released/offline devices alone. Check recovery_owner:"daemon", resolve hint/setup_blocked_on, and wait out any release/reconnect already in progress. Then send mode=agent once and poll status until drivable:true; stop recovery when the task no longer needs the phone. Repeated bootstrap requests obscure the USB, trust, DDI, or VPN blocker.
• Do not batch guesses. During discovery, keep one action between reads. Once a segment is understood, default to phone_run_steps or POST /agent/actions and combine up to 24 steps in one call instead of paying one model/tool round-trip per tap. Put semantic wait_for gates around page transitions and prefer tap_locator over coordinates or repeated labels. The daemon validates the complete batch first, holds one WDA control lock, and stops before every later step on the first failure. Fixed pause/after_ms values are capped at 3 seconds and are only animation settles, not proof that the right page appeared.
• Treat outcome_unknown as possibly executed. A 502/504 after dispatch can mean the phone acted but its acknowledgement was lost. Read elements or a screenshot before deciding whether to send the action again; never blindly replay text, scroll, back, payment, send, or delete actions.
• A reliable "reset to known state": shortcut home, then shortcut spotlight
  • text <app name> + key return to launch any app.

Vision fallback: when the AX tree is unusable

AX-first is the invariant; vision is a screen-scoped fallback. Two failure modes trigger it:

Mode A — tree too sparse (games, canvas/WebGL, custom-drawn UI). /agent/elements succeeds; judge its additive ax_stats block ({n, n_interactive, labeled_frac, coverage, container_only, max_depth}):

• unusable → go vision: n_interactive == 0 && container_only (e.g. the 1-element tree whose only row is the Application node — coverage ≈ 1.0 there is meaningless, never read coverage before those two gates).
• degraded → hybrid: n_interactive < 3, or labeled_frac < 0.3, or (coverage < 0.3 and not container_only). Use AX for the rows it has, vision for the rest.
• Otherwise usable: stay AX-only. Legitimately sparse screens exist (video player with one "Done" button); low n alone is not a trigger — zero usable targets for your current intent is.

Mode B — reading the tree kills the runner (KakaoTalk, issue #44: any AX hierarchy snapshot crashes on-phone WDA; recovery is a 1–3 min rebuild). The signal: /agent/elements returns 502 wda_source_failed / 504 wda_source_timeout twice on the same foreground app while GET /agent/screenshot still succeeds. Cache that verdict per app for the session — do not re-probe and pay the rebuild again.

The AX-free loop (you are the grounding model; no new infra):

1. GET /agent/screenshot → reason over the pixels yourself and pick a target.
2. Act with the existing coordinate actions only — tap / longpress / swipe/scroll with normalized [0,1] coordinates. They dispatch via W3C /actions and never resolve the hierarchy. Unsure of the point (< ~0.5 confidence)? Send nothing (that is not_sent, retry-safe): re-screenshot, crop the region to look closer, or scroll the target into view — then report with a screenshot if still lost.
3. Verify with a post-action screenshot (settle ~300–800 ms), never with an element read. HARD RULE for Mode-B apps: no ?return=delta, no element/label taps, no element wait_for gates — each one resolves the tree and takes the device down. The loop must be hermetically AX-free while such an app is foreground. (In Mode A return=delta is merely useless — a game's tree doesn't change; screenshot diff is the verifier there too.)

Degradation ladder on the existing outcome grammar:

Vision guesses are weaker evidence than AX labels: destructive targets (send / pay / delete / 2FA) keep their explicit-verification rules regardless of channel. Any successful AX read on a new screen flips you back to AX-first, and every successful vision sequence should be compiled into a flow per the next section — vision is how you discover a flow, not how you run it the tenth time.

Semantic intents (registered Shortcuts verbs)

When the task maps to a verb in GET /agent/intents (check once per session), prefer one semantic call over driving the UI: POST /agent/intent with {"name":"battery","args":{}} opens the bridge shortcut's deep link on-device and returns an id; the structured result lands on /agent/inbox (peek with GET, consume with POST /agent/inbox/drain, match on that id). The registry is deliberately small and human-curated — an empty list is the normal answer, and then you use the UI channel. Caveats: the Shortcuts app foregrounds during the run (never interleave with a mid-flight UI flow; re-orient after), and the first run of each verb needs a one-time interactive permission blessing on the phone — if a call dispatches but no inbox reply appears, a pending permission dialog on the phone screen is the first suspect. At-most-once rules apply unchanged: outcome:"not_sent" is retry-safe, outcome:"unknown" (intent_timeout/intent_dispatch_failed) means check the inbox and observe state before ever re-sending a side-effecting verb.

Self-improvement: vision once → script forever

The first time you do a task, you're vision-guided (screenshot + reasoning at every step). That's expensive. Your job is to never pay that cost twice:

1. While solving, log intent and evidence, not only the wire payload — record the intended target, the successful accessibility label/role, the precondition, the action, and the observed postcondition. A screenshot or snapshot element index is evidence from that moment, not a durable locator.
2. Compile the successful trace into a guarded flow. Prefer a fresh-resolved accessibility identifier, then a unique role + label + state, then a container/anchor relationship. Zero matches and multiple matches both fail closed. Never persist WDA element IDs, /agent/elements indexes, or snapshot tokens: they are valid only for the source read that produced them.
3. Coordinates are the final fallback, not a reliability claim. A normalized point can drift after an app update, A/B test, keyboard change, dynamic list reorder, orientation change, or a different phone. If a pixel-only action is unavoidable, bind it to a known screen signature and an immediate postcondition; refuse to run when those checks do not match.
4. Wait for states, not fixed sleeps. Poll a cheap element/status postcondition with a bounded timeout. Use a fixed delay only for a transition that has no observable state, and keep it short and explicit.
5. Keep checkpoints, drop repeated reasoning. The happy path should need no model and no screenshot tokens. On a failed checkpoint, stop and collect the last action, current elements, status, and one screenshot for repair. Patch the broken locator or branch and create a new flow revision; do not silently guess a replacement target.
6. Respect at-most-once delivery. Retry automatically only when the daemon proves outcome:not_sent and the action is still valid. For outcome_unknown, inspect current state before deciding. Never blindly replay text, scroll, back, payment, send, publish, comment, like, follow, or delete actions.
7. Keep secrets and user data out of v1 flow files. Explicit string inputs use {"kind":"type","input":"query"} and are resolved only for the current browser run or flow run --input query=value; the saved JSON never contains the value. CLI values may still appear in shell history or process inspection. Never use flow parameters for passwords, session tokens, one-time codes, private content, payment, send, publish, comment, like, follow, or delete actions.

The research and flow contract live in docs/scripted-flows-research.html. phone_run_steps is the bounded in-memory runner for stable segments; the release-matched iphone-use-mcp binary validates and runs version-1 JSON without a model, and flow update/list/info/ run <app>/<flow> mirrors the official registry into ~/.iphone-use/flows (see Registry first above). A compiled flow that works should end up in the registry: add app, category, risk, locale, and verified_on, then open a PR to leeguooooo/iphone-use-flows. There is still no branching or repai

…(truncated)