Macos Agent Ops

macos-agent ops

Contract

Prereqs:

• macOS host with Accessibility and Automation permissions granted for Terminal and target apps.
• Homebrew-installed macos-agent available on PATH.
• cliclick, osascript, and im-select available on PATH.
• ABC/US input source enabled in macOS Input Sources.
• For full AX surface (ax attr/action/session/watch), Hammerspoon runtime should be healthy (hs + hs.ipc).

Inputs:

• Command entrypoint: $AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh.
• Optional env vars:
  • MACOS_AGENT_OPS_INPUT_SOURCE_ID (preferred; example: abc) for deterministic input source target.
  • MACOS_AGENT_REAL_E2E_INPUT_SOURCE (legacy fallback) if MACOS_AGENT_OPS_INPUT_SOURCE_ID is unset.
  • MACOS_AGENT_OPS_SKIP_INPUT_SOURCE_SWITCH=1 to bypass automatic input source switch.
  • CODEX_MACOS_AGENT_AX_BACKEND=auto|hammerspoon|applescript to control AX backend preference.

Outputs:

• macos-agent JSON/text command output.
• Artifacts generated by macos-agent under $AGENT_HOME/out/ (screenshots, traces, debug bundles, step ledgers).

Exit codes:

• 0: success
• 1: runtime failure or missing runtime dependency
• 2: usage error

Failure modes:

• wait app-active timeouts because focus was stolen (Control Center/Spotlight/notifications).
• window.activate failure when app relaunch/update state is stuck; wrapper now uses --reopen-on-fail.
• missing im-select causes input-source failure before keyboard-input actions.
• not authorized / Apple Events denial from TCC permissions.
• AX target mismatch (selector too broad / stale node id) during ax operations.
• AX gate failures (--gate-*) when app/window/selector readiness conditions are not met.
• AX postcondition failures (--postcondition-*) when state does not change as expected after mutation.
• observe screenshot --if-changed-baseline path missing or unreadable.
• trace directory validation failures when using --trace/--trace-dir and path is not writable.

Scripts (only entrypoints)

• $AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh

Workflow

1. Resolve Homebrew macos-agent binary path:

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh where

1. Ensure deterministic input source (default target: abc):

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh input-source
# or explicit source id
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh input-source --id com.apple.keylayout.ABC

1. Run readiness checks (preflight + AX list smoke check):

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh doctor
# optional target for AX smoke
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh doctor --ax-app Arc
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh doctor --ax-bundle-id com.google.Chrome

1. Inspect unified permission state directly (screen_recording / accessibility / automation / ready):

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json preflight --include-probes

1. Run quick app foreground check with reopen recovery:

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh app-check --app Finder
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh app-check --app Arc --timeout-ms 15000
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh app-check --bundle-id com.google.Chrome

1. Run AX-only health check (AX tree probe):

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh ax-check --app Arc --role AXWindow
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh ax-check --app Arc --role AXTextField --title-contains Search

1. Use AX wait primitives before mutation to stabilize dynamic UI:

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json wait ax-present --app Arc --role AXButton --title-contains "Compose" \
  --match-strategy contains --timeout-ms 2500 --poll-ms 80

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json wait ax-unique --app Arc --role AXTextField --title-contains "Search" \
  --selector-explain --timeout-ms 2500 --poll-ms 80

1. Run robust mutating AX actions with gating + postconditions:

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json ax click --app Arc --role AXButton --title-contains "Play" \
  --match-strategy contains --selector-explain --reselect-before-click \
  --fallback-order ax-press,ax-confirm,frame-center,coordinate \
  --gate-app-active --gate-window-present --gate-ax-unique \
  --postcondition-focused true --postcondition-timeout-ms 2000 \
  --allow-coordinate-fallback

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json ax type --app Arc --role AXTextField --title-contains "Search" \
  --text "lofi" --clear-first --submit --paste \
  --gate-app-active --gate-window-present --gate-ax-present \
  --postcondition-attribute AXValue --postcondition-attribute-value "lofi" \
  --postcondition-timeout-ms 2000 --allow-keyboard-fallback

1. Use selector-frame screenshots for visual proof of target alignment:

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --role AXButton --title-contains "Play" --selector-padding 12 \
  --path "$AGENT_HOME/out/macos-agent-selector-$(date +%Y%m%d-%H%M%S).png"

1. Use diff-aware screenshots to avoid noisy artifacts when nothing changed:

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --path "$AGENT_HOME/out/macos-agent-state.png" \
  --if-changed --if-changed-threshold 2

1. Use one-shot debug bundle for triage artifacts:

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json debug bundle --active-window \
  --output-dir "$AGENT_HOME/out/macos-agent-debug-$(date +%Y%m%d-%H%M%S)"

1. Run routine scripted scenarios:

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh scenario --file /path/to/scenario.json

1. Pass through raw macos-agent commands with JSON errors + trace when needed:

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json --error-format json --trace \
  input click --x 200 --y 160

Screenshot-Based Triage Rules

• During runtime failure triage, the agent SHOULD capture an active-window screenshot before retry/remediation.
• For wait app-active and window.activate failures, screenshot capture SHOULD be first-line diagnostics.
• Screenshot artifacts MUST be written under $AGENT_HOME/out/ and SHOULD use timestamped filenames.
• The agent SHOULD include screenshot path + failing command + error message in the final diagnostic summary.

Recommended command (via the skill entrypoint):

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --path "$AGENT_HOME/out/macos-agent-failure-$(date +%Y%m%d-%H%M%S).png"

Common Errors And Prevention

• error: input source mismatch after switch attempt ...

  • Run .../macos-agent-ops.sh input-source --id abc and verify current is com.apple.keylayout.ABC or ...US.
  • Ensure im-select is installed: brew install im-select.
  • If you intentionally need non-ABC IME, set MACOS_AGENT_OPS_SKIP_INPUT_SOURCE_SWITCH=1.

• error: timed out waiting for app-active ...

  • Keep hands off keyboard/mouse while checks are running.
  • Close Control Center/Spotlight overlays and reduce notification interruptions.
  • Increase timeout (--timeout-ms 12000 or higher) for slow app transitions.

• error: ax.list ... timed out during wait ax-present / wait ax-unique

  • Run app-check first, then retry AX waits after app focus is stable.
  • Increase AX wait timeout/poll (--timeout-ms, --poll-ms) and narrow selectors.
  • Keep backend on auto unless you need to force Hammerspoon behavior (CODEX_MACOS_AGENT_AX_BACKEND).

• window activate failed ...

  • Wrapper already uses --reopen-on-fail; rerun once to allow quit/relaunch recovery.
  • For flaky apps (e.g., Spotify updater), kill stale relauncher and retry.

• not authorized / Apple Events failures

  • Re-run doctor and fix Accessibility/Automation in System Settings.

• selector returned zero AX matches or ambiguous matches

  • Narrow selectors and use --match-strategy exact|prefix|suffix|regex when contains is too broad.
  • Use --selector-explain and wait ax-unique before ax click/ax type.
  • Prefer --reselect-before-click in dynamic pages where node ids churn.

• gate failure (--gate-*) before mutation

  • Add/adjust readiness gates based on context (--gate-app-active, --gate-window-present, --gate-ax-present, --gate-ax-unique).
  • Increase gate timeout/poll for slower app transitions (--gate-timeout-ms, --gate-poll-ms).

• postcondition failure (--postcondition-*) after mutation

  • Verify you are asserting the correct target/attribute (AXValue, focus state, etc.).
  • Increase postcondition timeout on animation-heavy apps (--postcondition-timeout-ms).

• error: --if-changed-baseline path does not exist ...

  • Verify baseline path exists before running capture.
  • If baseline is optional, remove --if-changed-baseline and let current output file be baseline.

• trace directory is not writable (trace.write errors)

  • Use a writable custom path with --trace-dir.
  • Prefer $AGENT_HOME/out/... for reproducible artifacts.

• Homebrew macos-agent missing

  • Install and link with Homebrew: brew install macos-agent.

Practical Notes

• Prefer canonical flags in examples:
  • --window-title-contains (instead of legacy --window-name)
  • input type --submit (instead of legacy --enter)
• Keep --error-format json enabled in automation loops when you need machine-parseable failures.
• Use debug bundle first when a run fails and root cause is unclear.
• Parse preflight JSON permissions (screen_recording / accessibility / automation / ready) before mutating runs.

References

• skills/tools/macos-agent-ops/references/e2e-ref-01-preflight-focus.md
• skills/tools/macos-agent-ops/references/e2e-ref-02-finder-routine.md
• skills/tools/macos-agent-ops/references/e2e-ref-03-matrix-routine.md

Converted and distributed by TomeVault — claim your Tome and manage your conversions.