Browser Automation Core

Browser Automation Core

Shared model for the browser-automation cluster. The drivers all differ in what they touch and how long they live — keep these concepts consistent here so no spoke contradicts another.

1. The decision the cluster turns on: which surface?

Everything starts with one question — what are you actually driving? There are two surfaces, and they do not overlap:

                 ┌─ Web / DOM surface ─────────────┐      ┌─ Native OS surface ────────────┐
your target ──→  │ a page in a Chromium browser    │  OR  │ a desktop app's real UI         │
                 │ → selectors, accessibility tree │      │ → OS accessibility elements     │
                 │ → browser / webapp-testing      │      │ → peekaboo                      │
                 └─────────────────────────────────┘      └─────────────────────────────────┘

• Web/DOM — anything rendered in a browser. You target CSS selectors / ARIA refs inside the page. Drivers: browser (headless Chromium, debug-first) and webapp-testing (Playwright against a local dev server).
• Native OS UI — a real macOS application outside any browser (menus, windows, the Dock, a native dialog). You target OS accessibility elements by ID/query/coords. Driver: peekaboo.

Rule: decide the surface before anything else. A web page is never a peekaboo job; a native menu bar is never a Playwright job. Picking the wrong surface is the #1 way these tasks fail.

2. Session lifecycle (the second axis)

Sessions auto-start on first use (browser) and auto-clean on idle; tmux sockets and peekaboo caches you manage explicitly. Close what you open — stop servers, kill sockets, quit apps.

3. Recon-then-act: never guess a selector

Every spoke follows the same loop: observe the live state, then act on what you found.

• browser — the primary command already returns a screenshot + console + network in one shot; query console / network / failed to dig.
• webapp-testing — goto → wait for networkidle → screenshot / read DOM → act on discovered selectors. Inspecting before networkidle on a dynamic app is the classic pitfall.
• peekaboo — see --annotate produces an annotated UI map with element refs (B1, T2, …); target those refs, not raw coordinates, whenever possible.

Refs/snapshots are invalidated by navigation — re-snapshot after every goto or window change.

4. Targeting matrix

5. The debug-first evidence rule (cluster's defining stance)

Debugging visibility is on by default, not opt-in. Console logs, network requests, and errors should already exist when a problem appears — you don't enable logging after the failure.

Therefore: you cannot claim a page or UI "works" until you have looked at the rendered evidence. A change is verified only when you have (a) the screenshot of the actual rendered state, AND (b) a clean console + network read (no uncaught errors, no failed requests). Screenshots taken by any spoke must be surfaced (Read the PNG) so they're actually seen — an unseen screenshot proves nothing.

6. Conventions

• Headless by default. Run headless; only open a visible browser when the user says "show me", honoring $BROWSER (fall back to the system default).
• macOS permissions. peekaboo needs Screen Recording + Accessibility — confirm with peekaboo permissions before driving native input.
• TTY input safety. When sending to TUI agents over tmux, send the text and Enter as separate send-keys calls with a small delay; a fast text+Enter can be swallowed as paste.
• Least surface. One tab / one window / one element at a time; widen only when needed.

7. Shared guardrails

• Decide the surface (web/DOM vs native OS) before choosing a tool.
• Recon before action: discover selectors/refs from a live snapshot; never hardcode guesses.
• Evidence before claims: no "it works" without a seen screenshot + clean console/network.
• Headless by default; visible only on explicit request.
• Confirm macOS permissions before native control; treat any new permission as a stated change.
• Close sessions, stop servers, kill sockets, quit apps when the task ends.