Simdeck

SimDeck Agent Guide

SimDeck automates iOS Simulators and Android emulators. Use the CLI for automation and the browser UI for live human visibility. iOS works with NativeScript, UIKit, SwiftUI, React Native, Expo, and Flutter apps; Android live viewing uses the emulator -share-vid shared display surface plus SimDeck's native host H.264 encoder, with lifecycle, screenshots, logs, and UIAutomator hierarchy dumps also handled through ADB.

SimDeck uses one long-running local service. Run simdeck first; it starts or reuses the background service and prints the browser URL plus pairing code. Pass --open only when you want SimDeck to open the default browser itself. Pass -p <port> or --port <port> when the service should use a non-default port. Pass -a or --autostart when the service should also be registered as a macOS LaunchAgent.

Example response from simdeck:

{
  "ok": true,
  "pairingCode": "401974",
  "pid": 91285,
  "projectRoot": "-",
  "started": true,
  "url": "http://127.0.0.1:4310"
}

simdeck
simdeck --open
simdeck -p 4311
simdeck -a
simdeck pair # prints LAN/Tailscale pairing URLs, code, and iOS QR
simdeck service stop
simdeck service kill # stops all SimDeck services from any binary
simdeck service restart

Usually http://127.0.0.1:4310. Pass a simulator name or UDID to simdeck when the browser URL should open with ?device=<UDID>. Use simdeck -p 4311 when the default port is not the right fit. Use simdeck pair when a native iOS client needs to pair. It starts or refreshes the LaunchAgent-backed service, detects LAN and Tailscale IPv4 addresses, and prints a QR with a simdeck://pair URL. Normal service restarts preserve the token and pairing code; use simdeck service reset only when you need to rotate them. Use simdeck service kill or simdeck service killall when you need a clean slate across installed and source-checkout binaries.

Optionally, when a user might want to drive the current simulator from their phone, run simdeck link [sim] to print an https://app.simdeck.sh/open universal link that opens SimDeck Studio on iOS (or the launchpad in a browser) pre-targeted at the resolved simulator.

Always first run simdeck and open the reported URL in the in-app browser using the Browser tool if available.

If Browser Use is not available, only then use simdeck --open; it opens the default browser and may take focus away from the app.

Device and app

Start by choosing a project default device. simdeck use <UDID> stores the selection for the current workspace/CWD so later commands can omit the UDID. Explicit UDIDs, --device, SIMDECK_DEVICE, and SIMDECK_UDID still work for one-off overrides. Prefer short forms in agent loops, such as simdeck tap "Continue" and simdeck snapshot --format agent --max-depth 2 -i.

simdeck list
simdeck list --format json
simdeck use <UDID>
simdeck boot <UDID>
simdeck boot android:<AVD_NAME> --android-emulator-arg=-no-snapshot
simdeck shutdown
simdeck erase
simdeck core-simulator restart
simdeck install /path/to/App.app
simdeck install /path/to/App.ipa
simdeck install android:<AVD_NAME> /path/to/app.apk
simdeck launch com.example.App
simdeck uninstall com.example.App
simdeck open-url myapp://route
simdeck open-url https://example.com
simdeck toggle-appearance
simdeck camera sources
simdeck camera start com.example.App --file /absolute/path/to/camera.mov
simdeck camera start com.example.App --webcam
simdeck camera switch --placeholder
simdeck camera status
simdeck camera stop

simdeck list defaults to compact JSON for token-efficient agent selection. Use simdeck list --format json only when you need full paths and display metadata.

Build apps with project tooling.

Android devices use IDs like android:Pixel_8_API_36. simdeck list discovers AVDs from the Android SDK. SimDeck-owned Android boots use emulator gRPC screenshot streaming for live video, keep the emulator shared-video surface as fallback, and use --android-gpu host by default. Android browser streams default to software H.264 at 60 fps with a 960px long-edge cap so emulator gRPC frame production stays smooth. On macOS, managed boots use -qt-hide-window so the Qt render loop stays active without showing the native emulator window. Use simdeck service restart --android-gpu auto or --android-gpu swiftshader_indirect only when host GPU rendering is unstable. Optional user defaults live in ~/.simdeck/config.json. Supported keys include service.port, android.emulatorArgs, and android.disableAudio; explicit CLI/API boot arguments still win for one-off runs.

Fast agent inspection

Use targeted checks for test loops. describe is a diagnostic snapshot of the whole hierarchy. For verification, prefer the service APIs exposed by simdeck/test: action, query, waitFor, assert, selector tap, and batch.

simdeck describe
simdeck describe --format agent --max-depth 4
simdeck describe --format agent --max-depth 4 --interactive
simdeck snapshot --format agent --max-depth 4 -i
simdeck describe --format compact-json
simdeck describe --point 120,240
simdeck describe --source auto
simdeck describe --source nativescript|react-native|flutter|swiftui|uikit
simdeck describe --source native-ax
simdeck describe --source android-uiautomator
simdeck describe --direct
simdeck wait-for --label "Welcome" --timeout-ms 5000
simdeck wait --label "Welcome" --timeout-ms 5000
simdeck assert --id login.button --source auto --max-depth 8

The default source is native-ax, which is the fastest and most universal path for agents. Use --source auto when you want richer NativeScript, React Native, Flutter, SwiftUI, or UIKit inspector data before native accessibility fallback. Use --direct or --source native-ax for the private CoreSimulator accessibility bridge. Use --source android-uiautomator for Android emulator UIAutomator hierarchies. For Android IDs, describe uses uiautomator dump; use --format agent or --format compact-json the same way as iOS. Use --interactive or -i when an agent only needs controls and actionable framework nodes; SimDeck keeps ancestor context so the output is still navigable. Agent output labels nodes with refs such as @e3; reuse them with simdeck press @e3. snapshot, press, and wait are aliases for describe, tap, and wait-for.

Prefer selectors, coordinates only when needed. Selector taps go through the service and wait for the element server-side. Use --expect-id, --expect-label, or another --expect-* selector when the tap should also wait for the next screen before returning.

simdeck tap --id LoginButton --wait-timeout-ms 5000
simdeck tap --id com.apple.settings.screenTime --expect-id BackButton
simdeck tap --label "Continue" --element-type Button
simdeck tap 120 240
simdeck tap "Continue"
simdeck press @e3

For persistent app integration tests, use simdeck/test instead of shelling out repeatedly:

import { connect } from "simdeck/test";

const udid = "your-device-udid";
const simdeck = await connect({ udid });

try {
  await simdeck.launch("com.example.App");
  await simdeck.waitFor({ id: "login.button" }, { maxDepth: 8 });
  await simdeck.tap(0.5, 0.5);
  await simdeck.assert({ label: "Welcome" }, { maxDepth: 8 });
  const matches = await simdeck.query({ id: "account.name" });
  console.log(matches);
} finally {
  simdeck.close();
}

simdeck/test state-query helpers default to source: "native-ax" for fast agent control. Pass source: "auto" only when you intentionally want richer framework inspector trees before native accessibility fallback.

Use tree()/describe only when a test needs to print the whole UI for debugging. In a normal agent loop, do not fetch the full tree after every action; verify the specific element or text that proves the step succeeded.

Interact

simdeck tap 120 240
simdeck touch 0.5 0.5 --phase began --normalized
simdeck touch 0.5 0.5 --phase ended --normalized
simdeck touch 120 240 --down --up --delay-ms 800
simdeck swipe 200 700 200 200
simdeck swipe 200 700 200 200 --duration-ms 500 --pre-delay-ms 100 --post-delay-ms 250
simdeck gesture scroll-up
simdeck gesture scroll-down
simdeck gesture swipe-from-left-edge
simdeck gesture swipe-from-right-edge
simdeck pinch --start-distance 160 --end-distance 80
simdeck pinch --start-distance 0.20 --end-distance 0.35 --normalized --duration-ms 250 --steps 8
simdeck rotate-gesture --radius 100 --degrees 90
simdeck rotate-gesture --radius 0.12 --degrees 45 --normalized --duration-ms 250 --steps 8
simdeck type 'hello'
simdeck type --stdin
simdeck type --file message.txt
simdeck key enter
simdeck key 42 --duration-ms 500
simdeck key-sequence --keycodes h,e,l,l,o --delay-ms 75
simdeck key-combo --modifiers cmd,shift --key z
simdeck dismiss-keyboard
simdeck button software-keyboard
simdeck button home
simdeck back
simdeck button lock --duration-ms 1000
simdeck button side-button
simdeck button volume-up
simdeck button volume-down
simdeck button action --duration-ms 1000
simdeck button mute
simdeck button digital-crown
simdeck crown --delta 50
simdeck button left-side-button
simdeck button siri
simdeck button apple-pay
simdeck home
simdeck app-switcher
simdeck rotate-left
simdeck rotate-right
simdeck pasteboard set 'text'
simdeck pasteboard get

Use --stdin or --file for text with quotes, newlines, shell variables, or shell-sensitive characters.

Timing and batch

simdeck tap --label "Continue" --wait-timeout-ms 5000
simdeck tap --label "Continue" --expect-label "Done"
simdeck back
simdeck swipe 200 700 200 200 --pre-delay-ms 100 --post-delay-ms 250
simdeck button lock --duration-ms 1000

Prefer to use wait-for or assert in a batch to wait for UI state instead of fixed delays. sleep 500 in a batch waits 500 ms. Use sleep 0.5s or sleep --seconds 0.5 when you want to write seconds explicitly.

Use batch when steps are known; use discrete commands when a later step depends on parsing previous output.

simdeck batch \
  --step "tap --label Continue --wait-timeout-ms 5000 --expect-label Done" \
  --step "type 'hello world'" \
  --step "back" \
  --step "pinch --start-distance 0.20 --end-distance 0.35 --normalized"

Batch rules: one source (--step, --file, or --stdin); set the default with simdeck use <UDID> or keep <UDID> at batch level; ordered steps; fail-fast by default; --continue-on-error for best effort. Step commands: tap, back, wait-for, assert, swipe, gesture, pinch, rotate-gesture, touch, type, button, key, key-sequence, key-combo, sleep.

For JS tests, batch can combine action and verification without extra CLI process startup:

await simdeck.batch([
  { action: "tap", selector: { label: "Continue" }, waitTimeoutMs: 5000 },
  {
    action: "waitFor",
    selector: { label: "Continue Tapped" },
    timeoutMs: 5000,
  },
  { action: "assert", selector: { id: "fixture.status" } },
]);

For app-style flows, SimDeck can run a practical subset of Maestro YAML:

simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro

Evidence

simdeck screenshot --output screen.png
simdeck screenshot --with-bezel --output screen-bezel.png
simdeck screenshot --stdout > screen.png
simdeck record --seconds 5 --output screen-recording.mp4
simdeck record --seconds 5 --stdout > screen-recording.mp4
simdeck logs --seconds 30 --limit 200
simdeck chrome-profile
simdeck processes
simdeck stats
simdeck stats --watch
simdeck sample --seconds 3

Use screenshots for still evidence, --with-bezel when the device frame matters, and record for short MP4 screen recordings. Use stats for simulator app CPU, memory, disk write, network receive/send rates, connections, hang, and crash/termination signals. Use sample only when a short CPU stack capture is worth the extra pause. Prefer describe for token-efficient state dumps, if they have enough context.

Camera Simulation

Use camera start when a booted iOS simulator app needs a deterministic camera feed. The CLI talks to the SimDeck daemon, which owns the camera source and shared-memory writer, then injects the camera shim into the app with simctl launch and relaunches the bundle. Use camera switch to swap between placeholder, media, and webcam sources without relaunching while the daemon feed is alive.

simdeck camera sources
simdeck camera start com.example.App --file /absolute/path/to/feed.mov --mirror off
simdeck camera switch --placeholder
simdeck camera start com.example.App --webcam "FaceTime HD Camera"
simdeck camera stop

Media paths must be absolute. URLs are treated as video streams. Webcam support depends on macOS camera availability and permission for SimDeck.

Default loop

1. Start UI, list, simdeck use <UDID>, boot/select the device, open viewer if in-app browser available
2. Build with project tools; install and launch with SimDeck.
3. Use one describe --format agent --max-depth 4 to understand an unfamiliar screen.
4. Interact with selectors first; use coordinates only when needed.
5. Verify with waitFor/assert/query, not repeated full describe dumps.
6. Batch known flows; keep describe as a failure/debug artifact.

Optional inspector plugins

For a richer hierarchy, if user wants to opt-in

NativeScript Inspector

NativeScript apps can connect directly to the running server from JS and expose their view hierarchy plus raw UIKit backing views

import { startSimDeckInspector } from "@nativescript/simdeck-inspector";

if (__DEV__) {
  startSimDeckInspector({ port: 4310 });
}

The runtime connects to GET /api/inspector/connect as a WebSocket

React Native Inspector

React Native apps can expose their component tree and Metro dev-mode source locations with the inspector package:

import "react-native-simdeck/auto";
import "expo-router/entry";

Import it before expo-router/entry or AppRegistry.registerComponent(...). The auto entrypoint no-ops outside development, reads EXPO_PUBLIC_SIMDECK_PORT when present, and otherwise scans common SimDeck service ports. Use the manual startSimDeckReactNativeInspector(...) API when you need custom host/path/security options.