IOS Sim Screenshots

iOS Simulator Screenshots

Drives the iOS Simulator headlessly from Bash + cliclick + osascript. The model takes a screenshot, looks at it via the Read tool, computes pixel coordinates of the next button, clicks, and repeats. Works for any SwiftUI/UIKit app that doesn't require external services to navigate.

Prerequisites

• Xcode.app installed at /Applications/Xcode.app
• cliclick (brew install cliclick if missing — required for sending taps)
• xcode-select -p may point at CommandLineTools, which lacks simctl. Do not run sudo xcode-select -s … — instead prefix every xcrun / xcodebuild call with DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer. No sudo required.
• macOS Accessibility permission must be granted for whatever process runs cliclick (Terminal, iTerm, Claude Code). First call may silently fail until granted.

Workflow

1. Boot a simulator and open the Simulator app

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcrun simctl boot "iPhone 17 Pro"
open -a Simulator

boot is idempotent and returns "already booted" harmlessly.

2. Build the app for the simulator

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
xcodebuild -project YourApp.xcodeproj \
  -scheme YourApp -configuration Debug \
  -destination 'platform=iOS Simulator,name=iPhone 17 Pro' \
  -derivedDataPath /tmp/app-dd build

Do NOT add CODE_SIGNING_ALLOWED=NO if the app touches the Keychain. Without proper signing the simulator returns errSecMissingEntitlement (status -34018) on every keychain write, which silently breaks wallet creation, login, etc. The default "Sign to Run Locally" identity is enough — just omit the flag.

This is a long build; run with run_in_background: true and wait for the completion notification.

3. Install and launch

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcrun simctl uninstall booted com.example.app  # safe even if not installed
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcrun simctl install   booted /tmp/app-dd/Build/Products/Debug-iphonesimulator/YourApp.app
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcrun simctl launch    booted com.example.app

Sleep ~2–3 seconds after launch before the first screenshot — SwiftUI splash transitions take a moment.

4. Calibrate coordinates (do this once per session)

The image is in device pixels; clicks land in screen points. You need both.

osascript -e 'tell application "System Events" to tell process "Simulator"
  set p to position of window 1
  set s to size of window 1
  return (item 1 of p) & "," & (item 2 of p) & "," & (item 1 of s) & "," & (item 2 of s)
end tell'

Returns e.g. 1919,178,456,972 → window origin (1919, 178), size 456×972 (points).

Combined with screenshot dimensions (e.g. 1206×2622 for iPhone 17 Pro), derive:

• scale = image_width / window_width (e.g. 1206/456 ≈ 2.645)
• title_bar_pt = 28 (standard macOS title bar height)
• screen_x = window_x + image_x / scale
• screen_y = window_y + title_bar_pt + image_y / scale

Where image_x, image_y are pixel coordinates measured on the original screenshot, not on a downscaled preview.

5. Screenshot → look → click → repeat

# capture
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcrun simctl io booted screenshot /tmp/step.png

Then Read the PNG via the Read tool. Identify the button you want, read its pixel coords from the image, run them through the formula above, then click:

osascript -e 'tell application "Simulator" to activate'  # ensure focus
sleep 1                                                  # let activation settle
cliclick m:<x>,<y> dd:<x>,<y> du:<x>,<y>                 # move, mouse-down, mouse-up
sleep 2                                                  # let UI animate
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcrun simctl io booted screenshot /tmp/next.png

Save keepers to your screenshots folder with cp once verified.

Gotchas (learned the hard way)

• A single cliclick c: often misses SwiftUI buttons. Use the explicit move + mouse-down + mouse-up triple (m: dd: du:). c: works for some buttons (e.g. plain Button { … }) but fails on others (sheets, list rows). Default to the triple.
• The first click after Simulator regains focus may be eaten by macOS to activate the window. Always osascript -e 'tell application "Simulator" to activate' then sleep 1 before the first click of a sequence — or accept that the very first click may need to be re-sent.
• Image previews shown back to the model are downscaled. When the Read tool reports e.g. "displayed at 920x2000, multiply by 1.31 to map to original 1206x2622," do that math — clicking off the displayed coordinates puts you 30% off-target.
• Scale is asymmetric in practice. Window height includes the title bar; image height doesn't. Subtract a ~28pt title bar and use the width-based scale for both axes. Off-by-a-few-points is fine for fat buttons, fatal for tab bar items at the screen edge.
• Tab bar at screen edge. Bottom navigation lands very close to window_y + window_height. If the click silently goes nowhere, the cursor is below the window — nudge image_y up by 30–50px or shrink the assumed title bar.
• Keychain on sim needs proper signing. Symptom: app shows red text like Failed to save to Keychain (status: -34018) after the first action. Fix: rebuild without CODE_SIGNING_ALLOWED=NO; uninstall + reinstall (state from the broken install can stick around).
• xcrun simctl io booted screenshot always writes a PNG even if you give a .jpg extension — name accordingly.
• Don't sudo xcode-select -s. It changes a global setting, may prompt for a password you can't see, and isn't needed. DEVELOPER_DIR=… env-prefix every command instead.

Suggested screenshot set for a typical app README

1. Launch / splash (no interaction needed, just delay)
2. Onboarding entry screen
3. Main / home view (post-setup; create a fresh account if needed)
4. A primary action sheet (Send / Compose / Add — captures the core flow)
5. Settings or profile view (high signal, easy to reach via tab bar)

Three to five screenshots is usually enough for a README. Don't chase coverage of every screen — the cost of navigating compounds with each tap, and brittle clicks waste tokens.

Cleanup

The simulator stays booted between sessions, which is fine. To shut it down explicitly:

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcrun simctl shutdown booted

Build artifacts in /tmp/app-dd are safe to leave; macOS will clean /tmp on reboot.