Chrome Relay

Chrome Relay

Drives the user's real Chrome through a Chrome extension + local native host. Prefer it when logged-in browser state (auth cookies, sessions, installed extensions) matters.

Setup

1. Chrome extension
2. CLI:

   pnpm add -g chrome-relay
   chrome-relay install
   chrome-relay doctor
   

Verify CLI ≥ 0.7.0 — wait/get/batch/snapshot --diff landed there (0.6.0 brought the snapshot/@ref loop; ≥ 0.5.20 fixed a silent click bug on Radix/React-Aria UIs):

chrome-relay --version

The core loop

chrome-relay tabs                             # find or create a tab
chrome-relay navigate "https://kushalsm.com" --new   # background tab by default
chrome-relay snapshot --tab 1234 -i           # see the page: actionable elements get @refs
chrome-relay click @e12                       # act on refs — no --tab, no selector
chrome-relay fill @e14 "hello"
chrome-relay wait --text "Saved" --tab 1234   # block until the page reacts
chrome-relay snapshot --tab 1234 --diff       # print only what changed (~100 tokens)

Snapshot output is compact indented text (~1–15 KB for most pages) — read it directly, no jq needed:

- link "Hacker News" [ref=e4]
- textbox "Search" [ref=e41]: current value
- checkbox "Remember me" [checked, ref=e42]
- clickable "Open card" [ref=e88]        ← cursor-pointer div the AX tree missed

Refs carry their own tab. click @e12 acts on the tab that produced e12, never the active tab — safe while the user keeps browsing. A contradicting --tab errors with target_conflict.

Ref lifetime. Refs survive same-page DOM churn (cached backendNodeId, healed by role+name re-find when nodes are replaced) but die on real navigation. A dead ref returns error.code = stale_ref → re-run snapshot.

Interception. Ref clicks hit-test the point first: if an overlay / sticky header / modal owns it, you get error.code = click_intercepted naming the interceptor — dismiss it or scroll, then retry. The click was NOT delivered. fill/type skip this check (covered inputs are still writable).

Tool surface

Picking the right text tool

Element addressing — the fallback ladder

1. @ref from snapshot -i — default. Covers buttons/links/inputs, named content, cursor-pointer div-soup (the sweep), and shadow DOM.
2. CSS selector — when you know the selector statically and don't need a snapshot.
3. js probe → coordinate click — canvas internals and SVG chart segments (anonymous <path> elements have no DOM handle anywhere):

   chrome-relay js --tab 1234 "const r = document.querySelector('svg path').getBoundingClientRect(); return {x: r.x + r.width/2, y: r.y + r.height/2}"
   chrome-relay click --tab 1234 --x 312 --y 218
   

Don't poll — wait

A snapshot after every action wastes turns. The cheap loop on a changing page:

chrome-relay click @e12
chrome-relay wait --text "Saved" --tab 1234     # or wait <selector> / --url / --load
chrome-relay snapshot --tab 1234 --diff         # only the changes, refs included

Top gotchas

0. snapshot -i is for ACTING, not fact extraction. It prints ref-bearing elements only — non-interactive values (dashboard metrics, paragraph text, chart labels) drop out. Measured live: a Cloudflare Pages metrics page lost all its numbers under -i. To READ facts, use full snapshot, get text <target>, or a js projection.
1. type appends — it inserts at the caret. If the input had a value (autosaved draft, default text), clear it first via js or keys (Cmd+A then Backspace).
2. Refs die on navigation — stale_ref means the page changed under you; re-snapshot. Don't retry the same ref.
3. Coords go stale fast — read getBoundingClientRect, scroll/reflow, then click → you hit the wrong element. For autocomplete popups especially, use keyboard nav, not coord clicks.
4. Click "succeeded" but nothing happened — first diagnostic: document.elementFromPoint(x, y). If it returns a wrapper or form background, your coords are wrong. If it returns the right element but state didn't change, you're likely on chrome-relay <0.5.20 — upgrade.

More recipes: references/patterns.md Failure modes: references/troubleshooting.md

Operational guidance

• Don't give up early. A failing click is information, not a stop signal. Attach a document-level listener with capture:true and watch what fires:

  chrome-relay js --tab 1234 "
    ['pointerdown','mousedown','click'].forEach(t =>
      document.addEventListener(t, e => console.log(t, e.target.tagName, e.target.className), {capture:true})
    );
    return 'listening'
  "
  # do the action, then:
  chrome-relay console --tab 1234
  

• Don't echo secrets. When extracting tokens / API keys via js, write the result directly to a file. Never echo $TOKEN or interpolate into shell strings — it ends up in scrollback, logs, and tool transcripts.
• Redact network output. Request/response headers carry cookies, auth/CSRF tokens, account and project IDs. Never paste raw chrome-relay network output into chat, docs, issues, or commits — filter to the fields you need (url, status, timings) or redact headers first.
• Capture before irreversible actions (form submit, send message, account change). Save the screenshot path.

Guardrails

• Errors are structured: branch on relayError.code (stale_ref, click_intercepted, element_not_found, target_conflict, timeout), not on message text.
• If a flag is unclear, chrome-relay <command> --help is authoritative — these docs lag.