# Browser To Curl > Turn a live browser session into a fast headless monitor. Use when the user says "monitor this website / page / dashboard", "watch X for changes", "tell me when Y updates", "keep an eye on this", or wants a login-gated page polled cheaply. The agent drives the site once with Playwright MCP (open, look, log in), finds the real HTTP request behind the data, and generates a standalone curl_cffi Python script (real Chrome TLS fingerprint, persisted cookie jar, reverse-engineered/refreshing login, change-diff, hook notification) that replays that request forever without a browser. Runs with uv. - Skill: `chuk-development/browser-to-curl` (Agent Skill, multi-file: 5 files) - Install (CLI): `npx skillmds@latest add chuk-development/browser-to-curl` - Raw SKILL.md: https://api.skillmd.com/api/skills/chuk-development/browser-to-curl/raw - Safety review: pending (external: skill-scanner PASS, skillspector CAUTION) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: chuk-development (https://skillmd.com/u/chuk-development) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/chuk-development/browser-to-curl --- # browser-to-curl Goal: the agent should **not** babysit a browser to watch a page. It drives the site once with Playwright to learn the request, then emits a small `curl_cffi` script that replays that request headless, forever, and pings back on change. Slow, expensive, human-shaped (Playwright, once) → fast, cheap, machine-shaped (curl_cffi loop). Same pattern the shipped scrapers in `~/git/google` and `~/git/ebay` use; this skill applies it to whatever page the user names. ## Tooling rules (do not skip) - **Python via `uv` only.** Generated scripts have a PEP 723 header, so `uv run monitor_x.py` fetches `curl_cffi` itself. Never `pip install`, never bare `python`. - **curl_cffi, not requests** — `impersonate="chrome"`. This is baked into the generated script; the reason is in `references/curl_cffi.md`. Read it. - **Secrets in env, never in the file.** Passwords, API keys, bearer/refresh tokens go in the spec as `${ENV_VAR}` and live in the environment. The generated `.py` gets committed and copied between agents — nothing secret in it. - Write monitors into the project, not a temp dir. ## The workflow — run every step, in order ### 1. Drive the site once (Playwright MCP) - `browser_navigate` to the page the user named. - If it needs login: log in. Prefer the human doing it once (VNC / the user types credentials), or type known credentials with `browser_type`. Either way, get to the logged-in view that shows the data. - `browser_snapshot` to confirm you see the target data on the page. ### 2. Find the real request behind the data - `browser_network_requests` — list what the page fetched. Sort by size: the data is almost always the **biggest non-asset response**. Ignore analytics beacons (`gen_204`, `/log`, `/collect`). - Pick the request that **returns the data as JSON** (an XHR/fetch to an API), not the HTML document. That is the one to replay. If the data is only in server-rendered HTML, replay the page URL and extract with a `regex` rule. - Note its method, full URL (incl. query string), and the headers that matter: `authorization`, `x-api-key`, `x-csrf-token`, `x-requested-with`, `referer`, `content-type`, and any custom `x-*`. For a POST, capture the body. - **Before you commit to an endpoint, run the JS-isolation test** (see `references/curl_cffi.md`): fetch the URL with `page.request.get` inside `browser_run_code_unsafe`. If that raw fetch already has the data, curl_cffi will get it too. If it returns a stub while the page shows the data, that endpoint is JS-assembled — pick another or fall back to Playwright for it. When a replay later returns a smaller "stub" payload, that is a gate: bisect URL → headers → cookies → TLS, cheapest first (same reference). ### 3. Export the session (cookies incl. httpOnly) - Run `scripts/pw_export.js` via `browser_run_code_unsafe`. It returns the full cookie jar (`name -> value`) and the dominant `cookie_domain`. `document.cookie` is **not** enough — it misses the httpOnly session cookie, which is usually the one that authenticates you. ### 4. Reverse-engineer the login (so it self-heals) The cookies from step 3 will expire. Decide how the script gets fresh ones: - **Capture the login request.** Log out and back in (or watch the network at login) and grab the login POST — url, headers, body fields. Put it in the spec as `relogin`, with `${ENV}` for the username/password. - **Token refresh:** if the site uses a short access token + a refresh token, capture the refresh call instead (or as well) and put it in `relogin`. The monitor replays `relogin` whenever it hits a wall, then retries once. - If you genuinely cannot replay login (MFA, captcha every time), skip `relogin` — the monitor will fire a `needs_login` event so a human re-seeds cookies. Tell the user this is the case. ### 5. Set the wall markers List the strings that mean "logged out / blocked" in `walls` (a login-URL fragment, a JSON flag like `"authenticated":false`, `Just a moment`, `Pardon Our Interruption`). Without this the monitor reports the login page as a "change". Get these from what you actually saw when a session dropped, or from `references/curl_cffi.md`. ### 6. Write the spec and generate Write a capture spec (see `references/spec.example.json`) and generate: ```bash uv run scripts/gen_monitor.py my-spec.json # -> monitor_.py ``` Fields: `name`, `impersonate`, `requests`, `cookies`, `cookie_domain`, `extract`, `walls`, `relogin` (optional), `interval`, `hook`. `extract.kind`: `json` (dotted `path`, `[*]` fans a list out) · `regex` (`pattern`) · `text` (whole body) · `contains` (`needle` → bool). Prefer `json` on a stable field (ids, counts, a status) so cosmetic HTML churn does not look like a change. ### 7. Verify before trusting it ```bash ACME_USER=... ACME_PASS=... uv run monitor_.py --once -v ``` Check: HTTP 200, no wall, the extracted sample is the real data (not a login page), a `.state.json` baseline got written. Run it a second time → it must say `unchanged` and fire nothing. If it fires on an unchanged page, your `extract` is too broad (a timestamp/nonce in the signal) — narrow it. ### 8. Wire the trigger (how it crawls back to the agent) The monitor exits **10 on an event**, 0 on no change — a hook can branch on that. Pick the delivery: - **`file`** (default): appends a JSON line to `~/.claude/monitor-events.jsonl`. A backend / Claude hook tails it and wakes the agent. Best for the platform. - **`stdout`**: prints `@@MONITOR@@ {json}`. Best under `/loop` — the loop reads the marker. - **`webhook`**: POSTs the event JSON to a URL. - **`command`**: runs an argv with the event JSON on stdin. Then schedule the poll. Prefer `--once` on a timer over the built-in loop, so a crash can't silently kill the watch: - cron: `*/5 * * * * cd && uv run monitor_.py --once` - or the built-in loop `uv run monitor_.py` (foreground, honours `interval`). ## Pre-ship checklist (tick every item) - [ ] `uv` used, PEP 723 header intact — no manual pip/venv. - [ ] `impersonate="chrome"` (curl_cffi), not requests/httpx. - [ ] Replaying the **API/JSON** request, not scraping rendered HTML (unless forced). - [ ] Cookies exported via `pw_export.js` (httpOnly included), correct `cookie_domain`. - [ ] `extract` reads a **stable** field — re-running on an unchanged page fires nothing. - [ ] `walls` set — a dropped session is a `needs_login` event, not a false "change". - [ ] Login/refresh reverse-engineered into `relogin`, OR user told it needs manual re-seed. - [ ] Every secret is `${ENV}`, nothing sensitive baked into the `.py`. - [ ] `--once -v` verified: 200, real data, baseline written, second run `unchanged`. - [ ] Hook + schedule wired so a change actually reaches the agent. - [ ] Poll `interval` matches the data's real cadence (minutes, not seconds). ## Files - `scripts/gen_monitor.py` — spec → standalone `monitor_.py` (embeds the runtime). - `scripts/pw_export.js` — Playwright snippet to dump cookies + `cookie_domain`. - `references/spec.example.json` — a filled-in spec. - `references/curl_cffi.md` — the anti-bot lessons the runtime relies on.