Web Scraping Browser

Web Scraping with the Browser (Hermes browser_* + CDP)

Static fetching gets you the HTML the server sends. The browser gets you the DOM the user sees. This skill is the playbook for the second one.

When to Use

Reach for browser_navigate instead of a static fetch when any of these hold:

Escalation path — always cheapest first:

1. Static fetch / web_extract on the URL.
2. Result empty, or body is just <script> tags and an app shell? → browser_navigate(url) + browser_snapshot.
3. Snapshot shows the content? Extract via browser_console (§Procedure, step 3).
4. Content requires being signed in? → CDP-attached signed-in session (§Procedure, step 5).
5. Content is visual-only (canvas, chart image)? → screenshot + vision_analyze (§Procedure, step 8).

Do not start at step 4. The signed-in browser is the escalation of last resort, not the default.

Prerequisites

• Hermes browser tools (browser_navigate, browser_snapshot, browser_console, browser_click, browser_type, browser_press, browser_scroll, browser_get_images, browser_dialog, browser_cdp) must be available in the current session.
• For signed-in scraping: the browser-hub Chrome must be running — the persistent instance with real user profiles at ~\Chrome\UserData, exposed on CDP http://127.0.0.1:9222. Verify it is up before attempting auth-gated work.
• For Playwright-over-CDP: the on-demand playwright MCP must be enabled (enable script + reload if needed).
• Windows host is primary. Commands and paths assume PowerShell on Windows. Path ~\Chrome\UserData is the hub's Chrome user-data directory — do not repoint or duplicate it.

Procedure

Step 1 — Core loop: navigate → snapshot → locate → act → re-snapshot → extract

Every browser scrape follows the same five-beat loop:

navigate → snapshot → locate (@ref or selector) → act (click/type/press/scroll) → re-snapshot → extract

Concrete sequence — scrape a product listing behind a search form:

browser_navigate("https://shop.example.com")
→ returns compact snapshot:
    textbox "Search products" @e3
    button "Search" @e4
    link "Sign in" @e7

browser_type(@e3, "mechanical keyboard")
browser_press("Enter")

browser_snapshot(full: false)
→ heading "Results for 'mechanical keyboard'"
    link "Keychron K8 Pro — $89" @e12
    link "NuPhy Air75 — $109" @e13
    button "Next page" @e29

Rules of the loop:

• Every action invalidates refs. @e12 is only valid against the snapshot that produced it. After any click/type/navigate/scroll, re-snapshot before using refs again.
• browser_snapshot(full: false) (compact) for orientation and finding interactive elements; browser_snapshot(full: true) when you need to read page content from the accessibility tree.
• Prefer browser_press("Enter") in a focused search box over hunting for a submit button — one fewer ref to resolve.
• For bulk data extraction, don't parse the snapshot — the snapshot is for navigation. Extract with browser_console (Step 3), which returns clean JSON.

Step 2 — Verify the page loaded and rendered

Never assume load-complete after browser_navigate returns. Poll a condition before extracting:

Poll for an element (the workhorse):

browser_console("!!document.querySelector('.results-grid .card')")
→ false … re-issue after a beat … → true → proceed to extract

Poll for a data sentinel (element exists but is a skeleton/spinner):

browser_console(`
  (() => {
    const el = document.querySelector('.price');
    return el && !el.closest('[class*="skeleton"]') && /\d/.test(el.innerText);
  })()
`)

Network/loader idle via CDP:

browser_cdp("Runtime.evaluate", {
  expression: "document.readyState === 'complete' && !document.querySelector('[class*=\"spinner\"],[class*=\"loading\"]')",
  returnByValue: true
})

Cap every polling loop at ~10 attempts. If the sentinel never turns true: wrong selector, content is inside an iframe (Step 6), or the page errored — snapshot and look before retrying harder.

Dialogs: JS alert/confirm/prompt dialogs block everything until handled. If a snapshot or action stalls and the page has a pending dialog, resolve it:

browser_dialog(action: "accept")                       // or "dismiss"
browser_dialog(action: "accept", prompt_text: "yes")   // for prompt()
// equivalent raw CDP: browser_cdp("Page.handleJavaScriptDialog", { accept: true })

Step 3 — Extract data via the page (browser_console)

browser_console(expression) evaluates JS in the page and returns the serialized result. This is your primary extraction tool — it reads the live DOM, including everything JS built after load.

Golden rule: return plain JSON-serializable values. No DOM nodes, no functions, no circular refs. Always .map() down to strings/numbers/plain objects.

Pull a table:

browser_console(`
  JSON.stringify([...document.querySelectorAll('table.results tbody tr')].map(tr => {
    const c = tr.querySelectorAll('td');
    return {
      name:  c[0]?.innerText.trim(),
      price: c[1]?.innerText.trim(),
      link:  tr.querySelector('a')?.href ?? null
    };
  }))
`)

Pull an article (SPA where web_extract saw nothing):

browser_console(`
  JSON.stringify({
    title: document.querySelector('h1')?.innerText,
    byline: document.querySelector('[class*="author"], [rel="author"]')?.innerText ?? null,
    published: document.querySelector('time')?.getAttribute('datetime') ?? null,
    body: [...document.querySelectorAll('article p')].map(p => p.innerText).join('\\n\\n')
  })
`)

Pull all links matching a pattern:

browser_console(`
  JSON.stringify([...document.querySelectorAll('a[href*="/product/"]')]
    .map(a => ({ text: a.innerText.trim(), href: a.href }))
    .filter(x => x.text))
`)

Images: browser_get_images() returns {url, alt} for every <img> — use it before writing a custom console expression for images. Fall back to console only for background-image, <picture>/srcset, or lazy-load attributes:

browser_console(`
  JSON.stringify([...document.querySelectorAll('img[data-src], img[srcset]')]
    .map(i => i.dataset.src || i.currentSrc || i.src))
`)

Wrap complex returns in JSON.stringify(...) yourself — it guarantees serialization and survives odd objects (e.g., elements with custom toJSON). Escape newlines inside template strings you embed ('\\n\\n' as above).

Extraction sanity check: if the console expression returns [], the selector is wrong OR the content hasn't rendered yet (Step 4). Debug with a counting probe first: browser_console("document.querySelectorAll('table').length").

Step 4 — Infinite scroll & lazy load

Two patterns, escalating:

Pattern A — snapshot-count loop (simple feeds):

browser_navigate("https://feed.example.com")
loop (max ~15 iterations):
  count = browser_console("document.querySelectorAll('[data-testid=\"post\"]').length")
  browser_scroll("down")
  browser_scroll("down")
  newCount = browser_console("document.querySelectorAll('[data-testid=\"post\"]').length")
  if newCount == count for 2 consecutive iterations → stop (feed exhausted or rate-limited)
extract everything in one console call

Pattern B — scrollHeight plateau via CDP (precise, works when browser_scroll steps are too small):

browser_cdp("Runtime.evaluate", {
  expression: "window.scrollTo(0, document.body.scrollHeight); document.body.scrollHeight",
  returnByValue: true
})
→ { result: { value: 14200 } }

// poll: repeat the same call; when value stops growing across 2–3 polls, you've hit bottom
browser_cdp("Runtime.evaluate", {
  expression: "window.scrollTo(0, document.body.scrollHeight); document.body.scrollHeight",
  returnByValue: true
})
→ { result: { value: 14200 } }   // plateau → done

Between scrolls, give lazy content a beat by polling a sentinel rather than sleeping (Step 2). If the site uses a "Load more" button instead of true infinite scroll, treat it as pagination (Step 7): snapshot → click @ref of the button → re-snapshot → repeat until the button disappears.

Always extract after the loop, not per-iteration — one browser_console over the fully-loaded DOM is cheaper and dedupes itself. Exception: virtualized lists (DOM recycles rows as you scroll — count stops matching what you saw). For those, extract per-iteration and dedupe by a stable key (href, id) in your accumulator.

Step 5 — Auth & signed-in scraping

Signed-in scraping runs against the browser-hub Chrome — the persistent instance with real user profiles at ~\Chrome\UserData, exposed on CDP http://127.0.0.1:9222.

HARD RULES — non-negotiable:

• Attach, never launch. connectOverCDP("http://127.0.0.1:9222"), then browser.contexts()[0], then context.newPage(). Never chromium.launch(), never browser.newContext() — a fresh context has no cookies and defeats the entire point.
• Never create or add a Chrome profile. If Chrome shows an "Add profile" / "create profile" prompt, decline and stop — you attached wrong or the hub isn't running. Do not click through it.
• Google SSO chooser: when a site's "Sign in with Google" opens the account chooser, pick the identity per browser-hub rules for the task (flow vs grok identity) — never guess, never add an account.
• Never point a self-launched Chrome at ~\Chrome\UserData. The hub owns that directory. Pointing your own launch at it causes profile-lock corruption. Attach over CDP instead.

Playwright-over-CDP (via the on-demand playwright MCP — enable script + reload if needed):

browser = playwright.chromium.connect_over_cdp("http://127.0.0.1:9222")
ctx = browser.contexts[0]          # THE existing signed-in context — do not create a new one
page = ctx.new_page()
page.goto("https://app.example.com/dashboard")
data = page.evaluate("() => [...document.querySelectorAll('.row')].map(r => r.innerText)")
page.close()                       # close your page; never close the browser/context

Same thing with Hermes tools only (browser tools already ride the hub session): just browser_navigate("https://app.example.com/dashboard") — if the profile is signed in, you're in. Verify auth state before scraping:

browser_console("document.cookie.length > 0 && !location.pathname.includes('/login')")

Cookie reuse — export session cookies to replay requests statically (much faster for bulk API scraping once you have the session):

browser_cdp("Network.getAllCookies", {})
→ { cookies: [ { name: "session_id", value: "...", domain: ".example.com", ... }, ... ] }

Filter to the target domain, build a Cookie: header, and hit the site's JSON API directly with static fetches. This hybrid (browser for auth, static for volume) is the fastest pattern for large authenticated scrapes. Treat exported cookies as credentials: keep them out of committed files and logs.

Step 6 — Hard cases: Shadow DOM, iframes, captchas

Shadow DOM — querySelectorAll doesn't pierce shadow roots; walk them explicitly:

browser_console(`
  JSON.stringify((function collect(root, out = []) {
    root.querySelectorAll('*').forEach(el => {
      if (el.shadowRoot) collect(el.shadowRoot, out);
      if (el.matches('.item-title')) out.push(el.innerText.trim());
    });
    return out;
  })(document))
`)

Snapshots usually do surface shadow content (the accessibility tree pierces it), so refs from snapshots often still work for clicking — it's console extraction that needs the walker.

Iframes — same-origin iframes are reachable via document.querySelector('iframe').contentDocument. Cross-origin iframes are not; take the frame_id from the snapshot and target the frame directly:

browser_cdp("Runtime.evaluate",
  { expression: "JSON.stringify([...document.querySelectorAll('.row')].map(r => r.innerText))", returnByValue: true },
  frame_id: "<frame_id from snapshot>")

If you need the frame's own targets: browser_cdp("Target.getTargets", {}) and match by URL, then pass target_id.

Captchas / challenge pages — do not attempt to solve or bypass captchas programmatically. Legitimate options: (a) escalate to the signed-in browser-hub session, where an established profile + cookies usually means the challenge never fires; (b) navigate in the visible browser and let the human user clear it once, then continue in the same session; (c) stop and report if the site's ToS prohibits automated access. Respect robots/ToS — a challenge is the site saying no.

Step 7 — Pagination buttons

loop:
  browser_snapshot(full: false)          // fresh refs every iteration — mandatory
  extract this page via browser_console  // paginated DOMs replace content, so extract per-page
  find button "Next" @eNN in THIS snapshot
  if absent or disabled → done
  browser_click(@eNN)
  poll sentinel for new page content (Step 2 — e.g. first-row text changed)

Step 8 — Device/viewport emulation (mobile-only content)

browser_cdp("Emulation.setDeviceMetricsOverride",
  { width: 390, height: 844, deviceScaleFactor: 3, mobile: true })

Step 9 — Save with provenance

Every scrape ends with a provenance-stamped artifact. Extracted-but-unverified data is a liability.

Save JSON with source metadata, not bare arrays:

{
  "source_url": "https://shop.example.com/search?q=mechanical+keyboard",
  "scraped_at": "2026-07-15T14:32:00Z",
  "method": "browser_console querySelectorAll('.card')",
  "record_count": 48,
  "records": [ ... ]
}

Report honestly: if 3 of 48 records have null prices, say so in the artifact and the summary — don't silently drop them.

Pitfalls

1. Stale refs. @e12 belongs to one snapshot. Any navigate/click/type/scroll → re-snapshot before the next ref-based call. Symptom: "ref not found" or clicking the wrong element.
2. Scrolling without re-snapshotting. New content after scroll is invisible to your old snapshot. Scroll → snapshot → then act.
3. Launching Chrome for authenticated work. Signed-in scraping = attach to http://127.0.0.1:9222 + contexts()[0] + newPage(). launch() / newContext() = fresh, logged-out, fingerprint-suspicious browser.
4. Touching Chrome's default User Data dir with your own launch. The hub owns ~\Chrome\UserData. Never point a self-launched Chrome at it (profile-lock corruption); attach over CDP instead.
5. Clicking "Add profile" / "create profile" in Chrome. Never. That prompt means you're in the wrong place — stop and re-attach.
6. Non-serializable console returns. Returning DOM nodes/NodeLists/functions from browser_console yields {} or garbage. Always map to plain data and JSON.stringify it; escape embedded newlines ('\\n').
7. Extracting before render. [] from a correct-looking selector usually means "too early", not "no data". Poll a sentinel (Step 2); never assume load-complete after navigate returns.
8. Parsing the snapshot as your dataset. Snapshots are lossy, truncated, formatted for navigation. Bulk data comes from browser_console → JSON. Snapshot = eyes; console = hands.

Verification

Every scrape must pass at least the first two of these three checks:

1. Count check: does record_count match what the page claims ("48 results")? Read the claim off the page:

   browser_console("document.querySelector('.results-count')?.innerText")
   

   Compare the returned text to your record_count. Mismatch = selector drift or missing pages.

2. Sample cross-check: take 2–3 random records and confirm they appear in a fresh browser_snapshot(full: true) — this catches selector drift where you scraped the wrong nodes (e.g., sponsored placements instead of results).

3. Visual confirmation: screenshot the page and vision_analyze it against a sample of your data ("does this screenshot show a product 'NuPhy Air75' at $109?"). This is the only check that catches rendering mismatches — CSS-hidden elements, overlapped content — that DOM extraction happily scrapes anyway.

Quick verification commands:

Quick reference