Peekaboo

Peekaboo

CLI timing flags accept bare milliseconds or ms/s suffixes, for example 500, 500ms, 2s, or 1.5s. Coordinate input uses --at x,y; add --global when a targeted coordinate should remain screen-global.

Peekaboo is a macOS automation CLI and agent runtime. Prefer the freshly built repo binary, live help, and canonical docs over copied command references because command surfaces move quickly.

Peekaboo 4 uses press for standalone keys and chords, drag --from/--to for drags, see --tree for CLI AX inspection, action for named accessibility actions, and real noun/verb subcommand trees. Use verify for stable predicates instead of fixed sleeps. JSON action results include an effect value; read-only results omit it. See docs/v4-migration.md when converting older invocations.

Start Here

1. In repo work, build and use the current-source binary:

   pnpm run build:cli
   BIN="$(swift build --package-path Apps/CLI --show-bin-path)/peekaboo"
   "$BIN" --version
   

2. Record the installed PATH binary separately when diagnosing environment drift:

   command -v peekaboo && peekaboo --version
   

3. Confirm permissions and current tool surfaces before automation:

   "$BIN" permissions status --json
   "$BIN" tools --json
   "$BIN" learn
   

4. Find command docs:

   node scripts/docs-list.mjs
   

Canonical References

• Live CLI help: peekaboo <command> --help
• Full agent guide: peekaboo learn
• Tool catalog: peekaboo tools
• Command docs in this repo: docs/commands/README.md and docs/commands/*.md
• Permissions and bridge behavior: docs/permissions.md, docs/bridge-host.md, docs/integrations/subprocess.md
• Repo rules: AGENTS.md

Observation Strategy

• Use peekaboo see --tree --no-screenshot (CLI) or inspect_ui (MCP) for native macOS AX text, labels, buttons, text fields, control state, and element IDs when a screenshot would add noise.
• Use peekaboo see for screenshots, visual layout, annotated maps, pixels, colors, screen/menu-bar targets, or cases where AX text is missing or incomplete.
• Use browser for browser page content, forms, DOM/a11y snapshots, console, network, page screenshots, and performance traces when browser tooling is available.
• Use native Peekaboo tools for app chrome, browser toolbars, menus, dialogs, permissions, windows, and non-browser apps.
• Treat element IDs from see or inspect_ui as valid only for the current visible state. After a mutating action, verify from the action result or fetch fresh state.

Operating Rules

• Use peekaboo see --json --path /tmp/<name>.png or peekaboo see --tree --no-screenshot --json before element interactions so you have fresh element IDs and snapshot IDs.
• Prefer the exact element ID string from the current snapshot for clicks and typing; treat ID shapes as opaque. Use labels when IDs are unavailable and coordinates only as a last resort.
• Check peekaboo permissions status --json before assuming a capture or control failure is a CLI bug.
• Use --json when another tool or agent needs to parse results.
• Respect the user's desktop: avoid destructive app/window actions unless requested.
• If a command fails because the target UI changed, recapture with see before retrying.
• see --json element bounds are screen coordinates. Snapshot IDs keep element actions tied to the observed UI.
• When using see in agent smoke tests, pass an explicit /tmp --path so capture artifacts do not land in a user-visible default location.
• Prefer --foreground only when an app requires a key window, Space switch, or foreground mouse event. Background delivery is the default when Peekaboo can resolve a target process.

Common Workflows

# AX-only state when a screenshot would add noise.
peekaboo see --tree --no-screenshot --app Calculator --json > /tmp/peekaboo-calc-ax.json

# Visual layout plus element IDs and snapshot ID.
peekaboo see --app Calculator --path /tmp/calc.png --json > /tmp/calc.json
ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts j.dig("data","snapshot_id"); puts JSON.pretty_generate((j.dig("data","ui_elements")||[]).map{|e| e.slice("id","label","identifier","bounds")})'

# Click an element discovered in the current snapshot.
SNAP=$(ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts j.dig("data","snapshot_id")')
ELEMENT_ID="<element-id-from-current-snapshot>"
peekaboo click --on "$ELEMENT_ID" --snapshot "$SNAP" --json

# Browser page content and DOM-oriented actions belong to browser tooling.
peekaboo browser status --json

# Browser toolbar, menus, permission prompts, and native app chrome still belong to Peekaboo.
peekaboo menu list --app Safari --json

# Routine management examples stay read-only by default.
peekaboo clipboard get --json
peekaboo permissions status --all-sources --json
peekaboo app list --include-hidden --include-background --json
peekaboo config provider list --json
peekaboo agent sessions --json
peekaboo press cmd+shift+t --app Safari --window-id 12345
peekaboo verify --app Safari --window-exists --timeout 2s --json

Input Path Testing

Peekaboo has two broad input paths:

• UIAX/action path: accessibility actions such as AXPress, AXSetValue.
• Synthetic path: pointer/keyboard events, commonly the CAEvent/CGEvent-style path.

Useful overrides:

# Confirm command exposes the override.
peekaboo click --help | rg 'input-strategy|actionOnly|synthOnly'

# UIAX/action click path from a saved snapshot.
peekaboo see --app Calculator --path /tmp/calc.png --json > /tmp/calc.json
SNAP=$(ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts j.dig("data","snapshot_id")')
ELEMENT_ID="<element-id-from-current-snapshot>"
peekaboo click --on "$ELEMENT_ID" --snapshot "$SNAP" --input-strategy actionOnly --json --focus-background

# Direct accessibility action; good for proving UIAX independent of pointer events.
peekaboo action AXPress --on "$ELEMENT_ID" --snapshot "$SNAP" --json

# Synthetic click path; allow focus if you need visible app state to mutate.
peekaboo click --on "$ELEMENT_ID" --snapshot "$SNAP" --input-strategy synthOnly --json --foreground

# Negative control: coordinates cannot use actionOnly.
peekaboo click --at 10,10 --input-strategy actionOnly --json

Interpretation:

• actionOnly success proves live AX re-resolution and action invocation.
• synthOnly success proves coordinate resolution and event delivery, but verify app state independently.
• action AXPress is the cleanest UIAX smoke test.
• Compare with Computer Use or another AX inspector when labels/descriptions differ.

Calculator Smoke Test

Calculator is a handy fixture because it exposes descriptions and identifiers.

BIN="$(swift build --package-path Apps/CLI --show-bin-path)/peekaboo"
"$BIN" permissions status --json > /tmp/peekaboo-skill-refresh-permissions.json
"$BIN" tools --json > /tmp/peekaboo-skill-refresh-tools.json
"$BIN" see --tree --no-screenshot --app Calculator --json > /tmp/peekaboo-skill-refresh-calc-ax.json
"$BIN" see --app Calculator --path /tmp/peekaboo-skill-refresh-calc.png --json --timeout 10s > /tmp/calc.json
ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts JSON.pretty_generate((j.dig("data","ui_elements")||[]).select{|e| ["Clear","AllClear","One","Two","Add","Equals","StandardInputView"].include?(e["identifier"].to_s)}.map{|e| e.slice("id","label","identifier","description","help","bounds")})'

SNAP=$(ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts j.dig("data","snapshot_id")')
BUTTON_ID="<button-id-from-current-snapshot>"
"$BIN" action AXPress --on "$BUTTON_ID" --snapshot "$SNAP" --json
"$BIN" click --on "$BUTTON_ID" --snapshot "$SNAP" --input-strategy actionOnly --json --focus-background

Expected current behavior:

• see --json includes bounds for each ui_elements entry.
• see --tree --no-screenshot --json and see --json should expose element IDs plus Calculator identifiers such as One, Two, and StandardInputView. Copy the returned ID exactly instead of assuming a prefix or shape.
• tools --json should include browser, click, inspect_ui, and see.
• Snapshot-backed UIAX must use the captured app/window, not the frontmost app.

Repo Validation

node scripts/docs-lint.mjs
ruby -e 'h=File.read("skills/peekaboo/SKILL.md").split(/^---\s*$/,3)[1]; keys=h.lines.grep(/^[A-Za-z0-9_-]+:/).map { |line| line.split(":",2).first }; abort("frontmatter keys: #{keys.inspect}") unless keys.sort == ["description","name"]'
! rg -n 'elem_[0-9]+' skills/peekaboo/SKILL.md
! rg -n '^allowed-tools:' skills/peekaboo/SKILL.md
pnpm run build:cli
BIN="$(swift build --package-path Apps/CLI --show-bin-path)/peekaboo"; "$BIN" --version
"$BIN" click --help | rg -- '--foreground|--focus-background|--input-strategy|Opaque element ID'
"$BIN" see --help | rg -- '--json|--annotate|--app|--no-web-focus'
"$BIN" see --help | rg '\-\-tree|\-\-no-screenshot|\-\-app|\-\-json'
git diff --check -- skills/peekaboo/SKILL.md docs/agent-skill.md docs/commands/see.md docs/automation.md scripts/docs-lint.mjs

Notes:

• If tests fail with no such module 'Testing', record it as local toolchain fallout; still run builds/lint/live smoke tests.
• SwiftPM may warn about Commander identity conflicts; do not chase unless the task is dependency hygiene.
• Keep live validation artifacts under /tmp/peekaboo-skill-refresh-* and do not commit or publish screenshots or UI dumps.

Keep this skill compact. Do not vendor generated command references here; update canonical CLI docs or Commander metadata instead.