Argent Device Interact

Unified tool surface

All interaction tools below accept a udid parameter and auto-dispatch iOS vs Android based on its shape (UUID → iOS simulator, chromium-cdp-<port> → Chromium (CDP) app, anything else → Android adb serial). You use the same tool names on every platform.

Chromium (CDP) app = any Chromium runtime exposing a Chrome DevTools Protocol endpoint: an Electron app (boot it with boot-device + electronAppPath), or any Chromium-family browser (Chrome/Brave/Edge) launched with --remote-debugging-port. The latter is auto-discovered by list-devices on port 9222 plus anything in ARGENT_CHROMIUM_PORTS. The same describe/tap/swipe/keyboard/screenshot surface drives all of them.

Multi-tab / windows (Chromium only): a Chromium device may have several tabs / BrowserWindows. Use chromium-tabs to list them (stable ids t1, t2, …, optional labels), open a new one, select which is active, or close one. Every other tool (describe, gesture-tap, screenshot, debugger-evaluate, open-url, …) acts on the active tab, so chromium-tabs action=select before driving a different tab. Note: a cross-process navigation (some redirects) can swap a tab's underlying CDP target — re-run chromium-tabs action=list to pick it up under a fresh id.

Cookies & storage (Chromium only): chromium-cookies reads/writes cookies via the Network domain (so HttpOnly cookies are visible): action=get (optionally scoped by url), set (name, value, + url/domain, optional secure/httpOnly/sameSite/expires), delete (name), clear (all). chromium-storage reads/writes Web Storage for the active page: store=local|session, action=get (one key or all entries), set, remove, clear. Both are per-origin / active-tab. Handy for seeding auth before a flow or asserting app state after one.

TV targets (Apple TV / Android TV) are not covered by this skill. A TV target is focus-driven, not touch-driven — the gesture-* tools are the wrong tools for it. This applies to both Apple TV simulators (UUID-shaped, identical to iOS) and Android TV / leanback devices (serial-shaped, identical to a phone emulator). If list-devices tags your target runtimeKind: "tv", stop and use the argent-tv-interact skill: describe to read focus, tv-remote for remote / D-pad presses, and keyboard to type.

Physical iPhones (an iOS entry with kind "device") follow a different contract: automation is app-scoped and only a subset of these tools exists there. Use the argent-ios-device-interact skill instead.

For platform-specific caveats (Metro adb reverse, locked-screen describe errors, etc.), see § 9 Platform-specific notes at the bottom.

1. Before You Start

If you delegate simulator tasks to sub-agents, make sure they have MCP permissions. (In this bundle there is no MCP: call the tools through the parent skill's CLI translation — argent run <tool> --args '<json>'.)

Use list-devices to get a target id. Results are tagged with platform (ios, android, or chromium); booted/ready devices come first. Pick the first entry that matches the platform you need — if none are ready, call boot-device with udid (iOS), avdName (Android), or electronAppPath (boots an Electron app as a chromium device). A Chromium browser already running with a CDP port shows up directly — no boot-device needed. See argent-ios-simulator-setup / argent-android-emulator-setup for full setup flow.

Check tool flags before first use. (Upstream deferred-schema ToolSearch does not apply here.) Gesture tools (gesture-tap, gesture-swipe, gesture-pinch, gesture-rotate, gesture-custom) have numeric parameters — pass them as JSON numbers, never strings, or validation will coerce and reject them. Get each tool's exact flags with argent tools describe <tool> before calling it.

2. Best Practices

1. Before tapping, re-verify the target from a fresh describe (this replaces upstream's tapping_rule from an argent.md rules file, which is not bundled).
2. Before performing interactions, consider whether they can be dispatched sequentially - more on that in run-sequence.
3. Use gesture-swipe for lists/scrolling, not gesture-custom, unless you need non-linear movement. On Chromium use gesture-scroll instead — gesture-swipe is touch-only. Consider whether you need multiple swipes, if yes - use run-sequence. Pass momentum: false when the swipe should decelerate before ending for a precise movement.
4. Tap a text field before typing, then use keyboard to enter text.
5. Coordinates are normalized — always 0.0–1.0, not pixels.
6. For app navigation, use the element tree returned after each action (--- Elements after action (describe) ---); call describe only when no fresh tree is available for the current screen. It works on any screen without app restart. Do not navigate from screenshot pixels on regular in-app screens unless the tree failed to expose a reliable target. Use native-describe-screen only when you need app-scoped UIKit properties.

3. Opening Apps

Never navigate to an app by tapping home-screen icons. Use launch-app or open-url — they are instant and reliable.

launch-app — by bundle ID

{ "udid": "<UDID>", "bundleId": "com.apple.MobileSMS" }

Common IDs: com.apple.MobileSMS (Messages), com.apple.mobilesafari (Safari), com.apple.Preferences (Settings), com.apple.Maps, com.apple.Photos, com.apple.mobilemail, com.apple.mobilenotes, com.apple.MobileAddressBook (Contacts)

open-url — by URL scheme

{ "udid": "<UDID>", "url": "messages://" }

Common schemes: messages://, settings://, maps://?q=<query>, tel://<number>, mailto:<address>, https://... (Safari)

4. Choosing the Right Tool

5. Finding Tap Targets

IMPORTANT. When moved to a different screen after an action or do not know the coordinates of component, always perform proper discovery first.

Point follow-up native diagnostics after you already have a candidate point:

• native-user-interactable-view-at-point: deepest native view that would receive touch at a known raw iOS point; requires bundleId
• native-view-at-point: deepest visible native view at a known raw iOS point; requires bundleId

If describe Fails

Read the exact error and choose the action that matches it:

• Error mentions ax-service not available or daemon startup failure: the ax-service daemon could not start. Check that the simulator is booted. Use screenshot as a temporary fallback, or use native-describe-screen with an explicit bundleId if the app has native devtools injected.
• describe returns an empty element list: the screen may be blank, loading, or showing content without accessibility labels. Use screenshot to see what is visible, then retry after the content has loaded.
• describe succeeds but is not detailed enough for a React Native app: use debugger-component-tree next.
• You need app-scoped inspection with full UIKit properties (accessibilityIdentifier, viewClassName): use native-describe-screen with an explicit bundleId. This requires native devtools (dylib) injection.
• You already have a candidate point and want to confirm what would actually receive touch: use native-user-interactable-view-at-point. Use native-view-at-point when you want the visually deepest view instead of the hit-test target.

6. Tool Usage

gesture-tap — Single tap at a point

{ "udid": "<UDID>", "x": 0.5, "y": 0.5 }

Coordinates: 0.0 = left/top, 1.0 = right/bottom.

Before tapping near the bottom of the screen in React Native apps, check that "Open Debugger to View Warnings" banners are not visible — tapping them breaks the debugger connection. Close them with the X icon if present.

gesture-swipe — Straight-line gesture

{ "udid": "<UDID>", "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 }

Swipe up (fromY > toY) = scroll content down. Default duration: 300ms. Optional: "durationMs": 500 for slower swipe.

"momentum" defaults to true (a natural flinging swipe). Pass "momentum": false for a momentum-free swipe: the finger decelerates into the end point, resulting in little to no fling. It needs durationMs of at least 150 and is rejected below it.

gesture-pinch — Two-finger pinch

{ "udid": "<UDID>", "centerX": 0.5, "centerY": 0.5, "startDistance": 0.2, "endDistance": 0.6 }

All values are normalized 0.0–1.0 (fractions of screen, not pixels) — same as all other gesture tools. startDistance: 0.2 means fingers start 20% of the screen apart; endDistance: 0.6 means they end 60% apart. startDistance < endDistance = pinch out (zoom in). startDistance > endDistance = pinch in (zoom out). Defaults: angle: 0 (horizontal), durationMs: 300. Optional: "angle": 90 for vertical axis, "durationMs": 500 for slower pinch, "endCenterX"/"endCenterY" to let the centroid drift to a new center over the gesture (omitted = fixed center).

gesture-rotate — Two-finger rotation

{
  "udid": "<UDID>",
  "centerX": 0.5,
  "centerY": 0.5,
  "radius": 0.15,
  "startAngle": 0,
  "endAngle": 90
}

All positions and radii are normalized 0.0–1.0 (fractions of screen, not pixels). radius: 0.15 means each finger is 15% of the screen away from center. endAngle > startAngle = clockwise. Default duration: 300ms. Optional: "durationMs": 500 for slower rotation, and "radiusX"/"radiusY" (fractions of screen width/height; give both — they override radius) with radiusX·width = radiusY·height for a physically circular orbit — a single radius traces a physical ellipse on a non-square screen, coupling a slight pinch into the turn.

gesture-custom — Custom touch sequence

For long-press, drag-and-drop, and other complex sequences, see references/gesture-examples.md. Set "interpolate": 10 to auto-generate smooth intermediate Move events between keyframes.

button — Hardware button press

{ "udid": "<UDID>", "button": "home" }

Values: home, back, power, volumeUp, volumeDown, appSwitch, actionButton

keyboard — Type text or press special keys

{ "udid": "<UDID>", "text": "search query" }

One call does one action. text and key are mutually exclusive, and a call that carries both is rejected with nothing typed. To type and then submit, send two keyboard steps in one run-sequence (§ 8) — { "text": "search query" }, then { "key": "enter" }. Two separate calls do the same work, but cost an extra round-trip.

Special keys: enter, escape, backspace, tab, space, arrow-up, arrow-down, arrow-left, arrow-right, f1–f12. Optional: "delayMs": 100 between keystrokes (default 50ms) — applies to the iOS simulator and Chromium; it is ignored on Android phones/tablets (typed via adb input text, no per-key cadence), on Vega, and on TV targets.

Typing secrets. To enter a credential without its plaintext ever entering your context, transcript, or logs, use a secret placeholder in text (works in keyboard, paste, run-sequence keyboard steps, and flow type steps):

{ "udid": "<UDID>", "text": "{{secret:APP_PASSWORD}}" }

The placeholder is resolved on the machine running the tool-server, from the first of these that defines the name:

Rules:

• The result echoes the placeholder, never the value. An unknown name fails with the list of available secret names and every source it looked in, with paths — read that list before asking the user anything.
• The auto-screenshot after the call is skipped so the typed value cannot re-enter your context as pixels. Do not describe or screenshot a non-secure field you just filled with a secret — submit or navigate away first, then verify the resulting screen. To submit, put the text step and the Enter step in one run-sequence. The skip covers a whole batch that contains the placeholder, but a second bare keyboard call gets its own screenshot of the filled field.
• Nothing outside those sources is reachable; never ask the user to paste a secret value into the conversation. Ask them to put it in a secrets file instead — a file edit applies to the next call, while an exported env var only reaches a tool-server started afterwards.
• The project sources are found by walking up from the tool-server's working directory. If a project file is not being picked up, the failure's source list shows the paths actually consulted; ~/.argent/secrets.env needs no project and always applies.

paste — Paste text into the focused field

{ "udid": "<UDID>", "text": "482913" }

Puts text on the device clipboard (the host clipboard is untouched) and triggers the platform's paste shortcut. iOS simulator and Android emulator only; a TV target, a physical device, Chromium and Vega are rejected.

paste is not a faster keyboard. keyboard types the way a user types and stays the default for every text entry — a search query, a login, a form field. Reach for paste only where a real user would paste: a 2FA / OTP code copied from another app, a long link or token, or to test how the app handles pasted input. It also carries what keyboard can't type on a given platform (multi-line text, non-ASCII on Android), but that alone is not a reason to paste — ask whether the user would.

Tap the field first so it has focus; pasting with no focused field is a silent no-op, as with keyboard. text accepts the same {{secret:<NAME>}} placeholders as keyboard, with the same auto-screenshot skip.

rotate — Change orientation

{ "udid": "<UDID>", "orientation": "LandscapeLeft" }

Values: Portrait, LandscapeLeft, LandscapeRight, PortraitUpsideDown

await-ui-element — Block until a UI element reaches a state

Instead of polling screenshot/describe in a loop, use await-ui-element to block server-side until an element reaches an expected state (or timeoutMs, default 5000ms, elapses). It polls the same accessibility/DOM tree as describe. (For a plain pause, use your own harness sleep — this tool deliberately has no bare-timer mode.)

{ "udid": "<UDID>", "condition": "visible", "selector": { "text": "Continue" } }

• condition: exists, visible, hidden, or text.
• selector: { text?, identifier?, role? } — every provided field must match. text matches the element's label or value and role its element role (e.g. AXButton, button, TextView, StaticText), both as case-insensitive substrings; identifier matches its accessibility id / resource-id / testID exactly (case-insensitive), also accepting the unqualified Android resource-id name (submit matches com.example.app:id/submit). The synthetic ROOT container describe prints is never matched, so a role like AXGroup/html won't trivially "match the screen".
• Prefer a specific selector. A loose substring can match several elements, and the tool may then key off one you didn't mean: text reads the first visible match in reading order (top-to-bottom, left-to-right — the same order describe lists them, so it's the one you saw first; when no match is visible, the first match overall), while visible/exists are satisfied by any match. Disambiguate with a longer or more exact string, an identifier, or a role (e.g. pin to a text role like StaticText to skip a same-named button). On a text timeout the note quotes the matched element's text, so you can see which one it landed on.
• text condition also needs expectedText (substring the matched element must contain).
• hidden passes when the selector matches nothing. If note says it never matched, treat the check as failed and fix the selector. On iOS, a degraded empty tree does not report hidden success; the note gives the recovery hint.
• Optional timeoutMs (default 5000) and pollIntervalMs (default 400).

Returns { success, elapsed, note?, cause? }. On failure, note describes the result. cause is unmet, unreadable, or cancelled. Only unmet means the tree was readable and the condition was false.

await-screen-idle — Block until the screen stops changing

Use after launch/navigation and before a raw tap, when an early-painted element may still be moving:

{ "udid": "<UDID>", "timeoutMs": 3000, "minStableMs": 250 }

On local iOS, Android, and Chromium, the tool waits for a non-empty describe tree to stop changing. Continue only when settled: true. Pair it with a destination-specific await-ui-element; stillness does not identify a screen.

Use it only for live diagnosis. Do not record it or put it in run-sequence. Flows use await: { idle: true }, which also compares pixels. This live tool can return during a presentation-layer animation.

7. Screenshots

Use the explicit screenshot tool only when:

• You need the initial screen state before any action.
• You are about to edit visible UI and need a baseline capture before making changes.
• The auto-attached screenshot shows a transitional or loading frame.
• You require extra context.
• You want to check state after a delay (e.g. waiting for a network response).
• A permission dialog, system alert, or native modal overlay is visible and describe did not expose reliable targets.

When using screenshot for permission or native modal navigation:

• Do not switch to screenshot-driven navigation just because a modal is visible. On regular app screens and in-app modals, keep using describe.
• Prefer obvious, centered alert buttons such as Allow, OK, Don't Allow, Not Now, or Continue.
• Tap one control at a time and inspect the returned auto-screenshot before doing anything else.
• After the modal is dismissed, return to normal discovery with describe, native-describe-screen, or debugger-component-tree.

Prefer the dialog over the Settings tool. When the app triggers its own permission prompt, answering it here is the real user path — do that. Reach for the settings-permissions tool only when you can't get to the change through the app: pre-authorize/deny a permission before the app asks, re-enable one the user already denied (iOS won't re-prompt), or reset it so the prompt reappears. See the argent-settings-permissions skill.

Optional rotation parameter: { "udid": "<UDID>", "rotation": "LandscapeLeft" } — rotates the capture without changing simulator orientation.

Screenshots are downscaled by default (30% of original resolution) to reduce context size. Use the normal downscaled screenshot for UI context and state checks. scale accepts values from 0.01 to 1.0, but do not use scale: 1.0 as a general readability or tapping aid.

Use full-resolution screenshots only when saving baseline/current PNG files for comparison. In that case, suppress the image block so the full-size PNG is not loaded into agent context (through the CLI this means the tool prints the PNG's file path instead of attaching pixels):

{ "udid": "<UDID>", "scale": 1.0, "includeImageInContext": false }

For visual regression checks, before/after screenshot comparisons, and detailed screenshot-diff parameter guidance, use the argent-screenshot-diff skill. Keep this skill focused on device interaction mechanics and screenshot capture.

Troubleshooting

8. Action Sequencing with run-sequence

Use run-sequence to batch multiple interaction steps into a single tool call. Only one screenshot is returned — after all steps complete. Use cases: scrolling multiple times, typing and submitting automatically, known sequence of multiple taps, rotating device back and forth.

Do not use run-sequence when any step depends on observing the result of a previous step

Use cases

Use the sequencing when:

• Knowing that some action needs multiple steps without necessarily immediate insight of screenshot
• "scroll to bottom", "scroll to top", "scroll to do X" -> sequence scroll 3-5 times
• form interactions, "clear and retype field" -> you may use triple-tap to select all, type new value
• "submit form" → fill all fields in sequence, tap submit
• "go back to X" → defined tap sequence for the navigation

Allowed tools inside run-sequence

gesture-tap, gesture-swipe, gesture-scroll, gesture-drag, gesture-custom, gesture-pinch, gesture-rotate, button, keyboard, paste, rotate, shake, tv-remote, await-ui-element

The udid is shared — do not include it in each step's args. Optional delayMs per step (default 100ms).

Add an await-ui-element step to gate a later tap on a screen transition (e.g. tap → wait for the next screen's button → tap it). If its condition is not met before the timeout, the sequence stops at that step and the following steps do not run — so a mistimed tap can't fire against a screen that never settled.

Examples

Scroll down three times:

{
  "udid": "<UDID>",
  "steps": [
    { "tool": "gesture-swipe", "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } },
    { "tool": "gesture-swipe", "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } },
    { "tool": "gesture-swipe", "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } }
  ]
}

Type into a focused field and submit. This is the only way to mix text and a key, because one keyboard call cannot carry both:

{
  "udid": "<UDID>",
  "steps": [
    { "tool": "keyboard", "args": { "text": "hello world" } },
    { "tool": "keyboard", "args": { "key": "enter" } }
  ]
}

Tap a known button, then scroll down:

{
  "udid": "<UDID>",
  "steps": [
    { "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.15 } },
    {
      "tool": "gesture-swipe",
      "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 },
      "delayMs": 300
    }
  ]
}

Tap, wait for the next screen, then act on it — the await-ui-element step gates the tap after it:

{
  "udid": "<UDID>",
  "steps": [
    { "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.9 } },
    {
      "tool": "await-ui-element",
      "args": { "condition": "visible", "selector": { "text": "Continue" } }
    },
    { "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.5 } }
  ]
}

Prefer this over a fixed delayMs when a step depends on a screen transition: it adapts to real load time, and if the condition is not met before the timeout the sequence stops there so the next tap can't fire against a screen that never settled.

Stops on the first error (or unmet await-ui-element condition) and returns partial results.

9. Platform-specific notes

Android

• Metro reachability: run adb reverse tcp:8081 tcp:8081 on the device before the RN app starts, or Metro won't be reachable from the device. See argent-metro-debugger for the full workflow. Re-run if the device restarts.
• First-launch permission prompts: reinstall-app on Android always installs with -g so runtime permissions are pre-granted on first launch — no flag to pass.
• Locked screen / secure surfaces: describe throws a clear error if it can't capture (keyguard, DRM, Play Integrity). Unlock the device or fall back to screenshot.
• APK vs .app in reinstall-app: pass .apk absolute path on Android; .app directory on iOS.

iOS

(no iOS-only gotchas collected here yet — add them as they come up)