Browser To Curl

browser-to-curl

Goal: the agent should not babysit a browser to watch a page. It drives the site once with Playwright to learn the request, then emits a small curl_cffi script that replays that request headless, forever, and pings back on change.

Slow, expensive, human-shaped (Playwright, once) → fast, cheap, machine-shaped (curl_cffi loop). Same pattern the shipped scrapers in ~/git/google and ~/git/ebay use; this skill applies it to whatever page the user names.

Tooling rules (do not skip)

• Python via uv only. Generated scripts have a PEP 723 header, so uv run monitor_x.py fetches curl_cffi itself. Never pip install, never bare python.
• curl_cffi, not requests — impersonate="chrome". This is baked into the generated script; the reason is in references/curl_cffi.md. Read it.
• Secrets in env, never in the file. Passwords, API keys, bearer/refresh tokens go in the spec as ${ENV_VAR} and live in the environment. The generated .py gets committed and copied between agents — nothing secret in it.
• Write monitors into the project, not a temp dir.

The workflow — run every step, in order

1. Drive the site once (Playwright MCP)

• browser_navigate to the page the user named.
• If it needs login: log in. Prefer the human doing it once (VNC / the user types credentials), or type known credentials with browser_type. Either way, get to the logged-in view that shows the data.
• browser_snapshot to confirm you see the target data on the page.

2. Find the real request behind the data

• browser_network_requests — list what the page fetched. Sort by size: the data is almost always the biggest non-asset response. Ignore analytics beacons (gen_204, /log, /collect).
• Pick the request that returns the data as JSON (an XHR/fetch to an API), not the HTML document. That is the one to replay. If the data is only in server-rendered HTML, replay the page URL and extract with a regex rule.
• Note its method, full URL (incl. query string), and the headers that matter: authorization, x-api-key, x-csrf-token, x-requested-with, referer, content-type, and any custom x-*. For a POST, capture the body.
• Before you commit to an endpoint, run the JS-isolation test (see references/curl_cffi.md): fetch the URL with page.request.get inside browser_run_code_unsafe. If that raw fetch already has the data, curl_cffi will get it too. If it returns a stub while the page shows the data, that endpoint is JS-assembled — pick another or fall back to Playwright for it. When a replay later returns a smaller "stub" payload, that is a gate: bisect URL → headers → cookies → TLS, cheapest first (same reference).

3. Export the session (cookies incl. httpOnly)

• Run scripts/pw_export.js via browser_run_code_unsafe. It returns the full cookie jar (name -> value) and the dominant cookie_domain. document.cookie is not enough — it misses the httpOnly session cookie, which is usually the one that authenticates you.

4. Reverse-engineer the login (so it self-heals)

The cookies from step 3 will expire. Decide how the script gets fresh ones:

• Capture the login request. Log out and back in (or watch the network at login) and grab the login POST — url, headers, body fields. Put it in the spec as relogin, with ${ENV} for the username/password.
• Token refresh: if the site uses a short access token + a refresh token, capture the refresh call instead (or as well) and put it in relogin. The monitor replays relogin whenever it hits a wall, then retries once.
• If you genuinely cannot replay login (MFA, captcha every time), skip relogin — the monitor will fire a needs_login event so a human re-seeds cookies. Tell the user this is the case.

5. Set the wall markers

List the strings that mean "logged out / blocked" in walls (a login-URL fragment, a JSON flag like "authenticated":false, Just a moment, Pardon Our Interruption). Without this the monitor reports the login page as a "change". Get these from what you actually saw when a session dropped, or from references/curl_cffi.md.

6. Write the spec and generate

Write a capture spec (see references/spec.example.json) and generate:

uv run scripts/gen_monitor.py my-spec.json          # -> monitor_<name>.py

Fields: name, impersonate, requests, cookies, cookie_domain, extract, walls, relogin (optional), interval, hook.

extract.kind: json (dotted path, [*] fans a list out) · regex (pattern) · text (whole body) · contains (needle → bool). Prefer json on a stable field (ids, counts, a status) so cosmetic HTML churn does not look like a change.

7. Verify before trusting it

ACME_USER=... ACME_PASS=... uv run monitor_<name>.py --once -v

Check: HTTP 200, no wall, the extracted sample is the real data (not a login page), a .state.json baseline got written. Run it a second time → it must say unchanged and fire nothing. If it fires on an unchanged page, your extract is too broad (a timestamp/nonce in the signal) — narrow it.

8. Wire the trigger (how it crawls back to the agent)

The monitor exits 10 on an event, 0 on no change — a hook can branch on that. Pick the delivery:

• file (default): appends a JSON line to ~/.claude/monitor-events.jsonl. A backend / Claude hook tails it and wakes the agent. Best for the platform.
• stdout: prints @@MONITOR@@ {json}. Best under /loop — the loop reads the marker.
• webhook: POSTs the event JSON to a URL.
• command: runs an argv with the event JSON on stdin.

Then schedule the poll. Prefer --once on a timer over the built-in loop, so a crash can't silently kill the watch:

• cron: */5 * * * * cd <dir> && uv run monitor_<name>.py --once
• or the built-in loop uv run monitor_<name>.py (foreground, honours interval).

Pre-ship checklist (tick every item)

• uv used, PEP 723 header intact — no manual pip/venv.
• impersonate="chrome" (curl_cffi), not requests/httpx.
• Replaying the API/JSON request, not scraping rendered HTML (unless forced).
• Cookies exported via pw_export.js (httpOnly included), correct cookie_domain.
• extract reads a stable field — re-running on an unchanged page fires nothing.
• walls set — a dropped session is a needs_login event, not a false "change".
• Login/refresh reverse-engineered into relogin, OR user told it needs manual re-seed.
• Every secret is ${ENV}, nothing sensitive baked into the .py.
• --once -v verified: 200, real data, baseline written, second run unchanged.
• Hook + schedule wired so a change actually reaches the agent.
• Poll interval matches the data's real cadence (minutes, not seconds).

Files

• scripts/gen_monitor.py — spec → standalone monitor_<name>.py (embeds the runtime).
• scripts/pw_export.js — Playwright snippet to dump cookies + cookie_domain.
• references/spec.example.json — a filled-in spec.
• references/curl_cffi.md — the anti-bot lessons the runtime relies on.