Browser Automation with browser4-cli
Browser automation CLI for AI agents — Chrome/Chromium via CDP with accessibility-tree snapshots.
Invocation
The docs use browser4-cli as the generic command name. From within the Browser4
source tree use one of the following:
| Shell | Command | Notes |
|---|---|---|
| PowerShell (Windows) | ./b4w.ps1 <command> |
Primary dev wrapper; builds from source if needed |
| Git Bash (Windows) | ./b4w.sh <command> |
Quotes args automatically for pwsh safety |
| Git Bash (alt) | pwsh ./b4w.ps1 <command> |
Direct PowerShell invocation |
| Linux / macOS | ./b4w.sh <command> |
Same script works cross-platform |
| Any (installed) | browser4-cli <command> |
After browser4-cli install |
Important: The
$(./b4w.ps1) <command>syntax shown in some task instructions does not work in bash —$(…)is command substitution, not invocation. Usepwsh ./b4w.ps1 <command>or./b4w.sh <command>instead.
1. Core Loop
⚡ First-run latency: From a source tree, the first launch builds the runtime bundle via Maven (~1–3 min, before the spinner appears) and then starts the Browser4 backend (Spring Boot + JVM, ~10s). Subsequent commands are instant — the server stays alive between invocations. The spinner shows stage-level progress (JVM → Spring Boot → MCP tools) so you can see what's happening.
🖥️ Headless mode is the default for AI agents: Always open browsers in headless mode (
--headless) unless the user explicitly asks to see the browser window (e.g., "show me the browser", "open visibly", "I want to watch", or "headed"). Headless mode is faster, uses fewer resources, and avoids unnecessary GUI windows. Use--headedonly when the user specifically requests a visible browser. See the Display Mode section below (§2 Key Concepts) for details.
Every browser4-cli session follows this pattern.
1. OPEN browser4-cli open --headless <url> # headless by default for AI agents
browser4-cli goto <url> # or goto to navigate within existing session
2. SNAPSHOT browser4-cli snapshot -v 0 # capture accessibility tree (viewport 0 = current visible screen)
3. INTERACT browser4-cli click <ref> # use refs from the snapshot
browser4-cli fill <ref> <value>
browser4-cli press Enter
4. RE-SNAPSHOT browser4-cli snapshot -v 0 --auto-diff # verify what changed (diff vs previous)
5. EXTRACT browser4-cli htmlsnapshot get ... # or eval, or X-SQL (see §4)
Copy-Paste Template
browser4-cli open --headless "https://example.com" # headless by default for AI agents
browser4-cli snapshot -v 0 --stdout # read the page; note refs
browser4-cli fill <ref> "<value>" # interact
browser4-cli press Enter
browser4-cli wait --load networkidle
browser4-cli snapshot -v 0 --auto-diff --stdout # verify what changed
browser4-cli htmlsnapshot get text "<css-selector>" --all
For quick inline viewing without opening a file, add --stdout:
browser4-cli snapshot -v 0 --stdout # print snapshot to stdout instead of file
2. Key Concepts
Element Refs
After commands that modify browser state, browser4-cli saves an accessibility-tree snapshot — a YAML file showing the page structure:
- generic [ref=e7]:
- link "News" [ref=e191]:
- /url: https://example.com/news
- textbox "Search query" [ref=e35]
- button "Search" [ref=e25]
Each interactive element has a ref (e5, e12) — the element's Chrome DevTools Protocol backend node ID, prefixed with e (so e12345 refers to backend node 12345). Use them to target elements in click, fill, type, get attr, etc.
Note:
/urlfields may be relative (e.g./url: news,/url: from?site=example.com). The snapshot output includes the page URL at the top for resolution. For absolute URLs, usehtmlsnapshot get all attr "a[href]" hrefwhich returns full URLs after redirect resolution.
Ref Lifecycle
Refs are ephemeral — treat them as single-use handles. Any interaction can leave you with stale refs if the page re-renders or Chrome remaps backend nodes:
- Always re-snapshot after interactions:
click,fill,type,press,check,uncheck,select,hover,drag,dblclick. - Definitely re-snapshot after page/context changes:
goto,reload, tab switches, or clicks that navigate/update the page. - If you are chaining form actions: rely on the automatic post-action snapshot, then use refs from that fresh snapshot for the next step.
In practice, the safest loop is interact → re-snapshot → use new refs. This is the CLI's current guidance and avoids intermittent stale-ref failures on reactive pages.
Interaction commands capture an automatic snapshot after execution. Pass --no-snapshot to skip it when you plan to capture a fresh snapshot manually (saves a round-trip).
Output Modes
- Default — human-readable output on stdout.
--show-tip/-tip— show a relevant, rotating tip on stderr after each successful command. Tips are suppressed by default; use this flag to enable them.--json— single-line JSON envelope on stdout for commands that support structured output. This is the clean machine-readable mode for commands such astab-list,htmlsnapshot get,htmlsnapshot query, andeval. Exception:snapshotremains YAML-focused and warns on stderr instead of returning JSON snapshot data.- File output (default for AI commands) —
extractandsummarizesave their result to a timestamped file in.browser4-cli/snapshot/and print only a link; add--stdout(or--raw) to print the payload directly. Whenextract --schemais used, the requested schema fields are emitted as plain top-level JSON (in the file and on stdout) — no envelope to parse. --quiet/-q— suppress all normal output; only errors appear on stderr.
Display Mode (Headless vs Headed)
Browser4 can launch Chrome in two display modes:
| Mode | Flag | Window | Use case |
|---|---|---|---|
| Headless | --headless |
No GUI window | Default for AI agents — scraping, automation, CI/CD, server environments |
| Headed | --headed |
Visible browser window | Debugging, user demonstration, interactive development |
Choosing the whole setup (session × display × browser source — including when to attach to your own browser instead of launching one): see browser-modes.md.
Rule for AI agents: always use --headless by default. Headless mode is faster, uses fewer resources, and avoids cluttering the user's desktop with browser windows. The only reason to use --headed is when the user explicitly requests a visible browser — look for phrases like "show me the browser", "I want to see", "open visibly", "headed", "watch what happens", or "debug visually". If user participation in the page interaction is required — e.g., logging in or entering a verification code (CAPTCHA) — open a headed browser so the user can see and act on the page.
Set the display mode with the open command when starting a new session. The goto command does not accept --headless/--headed directly — it inherits the session's existing display mode:
browser4-cli open --headless https://example.com # headless (preferred default)
browser4-cli open --headed https://example.com # headed (only when user asks)
Once a session is open, use goto for subsequent navigations (the display mode persists):
browser4-cli goto https://other-page.com # stays headless (or headed) as set by open
Note —
gotoon first invocation: Whengotois the very first command (no prioropen), it auto-opens a new session using the CLI's default display mode, which is headless. The display mode is still fixed at session creation, so if you want a visible window, start withopen --headedbefore usinggoto.
Note — reconnecting to an existing session: The
--headless/--headedflags only take effect when creating a new session. Whenopenreconnects to an already-running session, the display mode is already set and the flags are ignored — the CLI prints a warning on stderr and the reconnect message shows the tab count so inherited state is visible. To change the mode of an existing session, close it first (close), thenopen --headlessto create a new one. To discard a stale session's tabs/cookies/location entirely and start clean, useopen --fresh(closes the current session, then opens a new one).
Sessions
Named sessions isolate browser state (cookies, localStorage, tabs) in a dedicated browser profile directory keyed by the session id — reopening a named session always restores the same profile instead of rotating through a shared pool. Use -s <name> to target a named session. goto auto-opens/reconnects — you rarely need to manage sessions manually.
Concurrent runs — always pass
-s <name>: the unnamed DEFAULT session is a singleton shared by every invocation that omits-s, andgotosilently reconnects to it (last writer wins). Two parallel agent processes that both omit-swill navigate each other's pages out from under one another. Parallel/automated runs should each use their own-s <name>(e.g.-s job-42) for isolated state.
The list command displays a "Next open" column showing what happens when goto or open targets a named session that already exists:
- Reuse — reconnects to the existing browser window (session is active on the backend).
- Refresh — opens a fresh window (session is stale or missing).
Session state is stored in ~/.browser4 by default. When that directory is not
writable (e.g. sandboxed shells), the CLI automatically falls back to
./.browser4-cli-state (workspace-relative) and prints a warning — set
BROWSER4_CLI_STATE_DIR to an explicit writable path to silence it.
BROWSER4_RUNTIME_DIR likewise overrides the runtime bundle location.
Configuration
The config command manages persistent CLI defaults stored in
~/.browser4/config.json (honours BROWSER4_CLI_STATE_DIR). These are global
fallbacks — an explicit flag or environment variable always wins per invocation.
| Key | Purpose | Overridden by |
|---|---|---|
server |
Default Browser4 server URL | --server / BROWSER4_CLI_SERVER |
timeout |
Default HTTP timeout (seconds) | --timeout |
proxy |
Default download proxy URL | --proxy |
session |
Default session name | -s / --session / BROWSER4_CLI_SESSION |
browser4-cli config # List all values + config file path
browser4-cli config list # Same as above
browser4-cli config get server # Print one value ("(not set)" if unset)
browser4-cli config set server http://localhost:8182
browser4-cli config set timeout 45 # Positive integer seconds
browser4-cli config delete session # Reset a key to its default
Notes:
config get/set/deleteuse the spaced form (config get server), notconfig-get server.timeoutmust be a positive integer;0and unknown keys are rejected with a non-zero exit.config set serversets the persistent default; a later--serverflag orBROWSER4_CLI_SERVERstill overrides it for that invocation.
Tab Management
Tab commands scope to a session — all operations affect the session targeted via -s <session> (or the DEFAULT session when -s is omitted).
Tab Lifecycle
1. LIST browser4-cli tab-list # See all tabs: index, GUID, title, URL
2. CREATE browser4-cli tab-new [url] # Open a new tab (about:blank if URL omitted)
3. SWITCH browser4-cli tab-select <index> # Switch by index
browser4-cli tab-select --guid <guid> # Switch by stable GUID
4. CLOSE browser4-cli tab-close <index> # Close by index
browser4-cli tab-close # Close current tab
browser4-cli tab-close --guid <guid> # Close by GUID
5. VERIFY browser4-cli tab-list # Confirm state after changes
Key notes
- GUIDs:
tab-listshows aGUIDcolumn. Use--guidfor stable targeting across tab reordering. Extension sessions show achrome:prefix on numeric GUIDs; regular sessions use 32-char hex GUIDs. - Machine-readable output: Use
--jsoneither before or after the command:browser4-cli --json tab-listorbrowser4-cli tab-list --json. Output is a JSON envelope:{"command":"tab-list","output":{"count":N,"tabs":[{"index":0,"guid":"...","url":"...","title":"..."}]},"status":"ok"}. Thetabsarray andcountare nested insideoutput. - Session scoping: Prefix tab commands with
-s <session-id>to target a non-default session. Thelistcommand shows all tracked sessions and their IDs. - Last-tab behavior: Chrome requires at least one open tab. Closing the last tab silently creates a replacement
about:blank—tab-listwill still show 1 tab afterward. - Tab insert position: New tabs are inserted by Chrome (not Browser4). The position depends on Chrome's native behavior which varies by platform, Chrome version, and mode — on Windows headless CDP, new tabs appear at index 0 (before the active tab); on macOS and some other configurations, they appear after the active tab. Always run
tab-listafter creating new tabs to confirm positions before switching by index. - No auto-snapshot:
tab-listandtab-closedo NOT trigger automatic snapshots. Aftertab-select, runsnapshotexplicitly to get fresh element refs for the new active tab. - Re-snapshot after switches:
tab-selectchanges the active page context. Capture a fresh snapshot before interacting with page elements in the new tab. - Extension sessions: When closing tabs on extension-attached sessions, the backend may report an error even though the tab was successfully closed (Chrome's
chrome.tabs.removecallback can fire an error after the tab is already gone). The CLI verifies that the tab was actually removed and treats the operation as successful in this case. Extension sessions may also show "Stale" inlistoutput after all tabs are closed — the session can be reconnected withattach --extension. - Extension re-attach creates a fresh tab scope: Each
attach --extensionestablishes a new WebSocket connection and creates its own tab tracking scope. After re-attaching (e.g., after navigating tochrome://version/which drops the connection), only tabs created through the new connection are visible intab-list. Tabs from the previous connection are still open in Chrome but are not tracked by the new session. To work with those tabs, either re-open them viatab-newin the new session, or use-s <name>to preserve a named session that survives re-attach.
Examples
# List all tabs in the default session
browser4-cli tab-list
# Machine-readable tab data (both forms work)
browser4-cli --json tab-list
browser4-cli tab-list --json
# Output: {"command":"tab-list","output":{"count":1,"tabs":[{"index":0,"guid":"...","url":"about:blank","title":"(no title)"}]},"status":"ok"}
# Open a tab and switch to it
browser4-cli tab-new https://httpbin.org/get
# Output:
# Created tab with GUID: 2AAA0C47... (https://httpbin.org/get)
# Switched to tab 0 (https://httpbin.org/get)
# Note: tab index varies by platform (0 on Windows headless; may appear
# after the active tab on other platforms). Run `tab-list` to verify.
# Close by GUID (survives reordering)
browser4-cli tab-close --guid 2AAA0C47D288D3943BA85D31AA8D084C
# Cross-session tab operations
browser4-cli -s ext-session tab-list
browser4-cli -s ext-session tab-new https://example.com
browser4-cli -s ext-session tab-select 0
3. Command Map
| Command family | Purpose | When to use | Full reference |
|---|---|---|---|
goto, open, close, reload |
Navigation & session management | Every session starts here | — |
snapshot |
Capture accessibility tree (AXTree) with element refs | Page structure & interaction — find elements to click, fill, etc. Use snapshot when you need refs (e5, e36) to interact with. |
snapshot.md |
snapshot grep |
Search snapshot content with regex | Find elements by text or pattern | — |
click, dblclick, drag, hover, fill, type, press, select, check, generate-locator |
Page interaction | Form filling, button clicks, mouse actions, navigation. type --method auto|chars|exec (needs a target ref) bulk-inserts long (>150 chars) or multi-line text in one execCommand('insertText') instead of typing per character |
— |
upload <ref> <file> [file...] |
Upload local files to a page file input | Send attachments/photos/documents to an <input type="file">; the target must be a file input, paths must be readable on the machine running the browser |
upload.md |
dialog-accept, dialog-dismiss |
Native JS dialog handling | After clicking buttons that trigger alert/confirm/prompt | — |
htmlsnapshot get, get all |
Extract text/html/attr via CSS selectors from stored HTML | Page content & text extraction — get article text, headings, attributes. Use htmlsnapshot when you need to read or extract page content. |
htmlsnapshot.md |
get <mode> <selector> [name] |
Live-DOM single-element read (text, html, box, styles, property, attr) — capture-free, works on the current live page with refs or CSS selectors |
Post-interaction verification & quick reads — "did the submit work?" without a capture round-trip. get text <sel> after a click/fill; get attr <sel> href for link URLs. Caveat: attribute/property reads via CSS are the least reliable mode, and a missing attribute reads as null (ambiguous with a failed resolution) |
— |
htmlsnapshot query |
X-SQL queries for structured extraction | Multi-field, filtered, sorted data | x-sql.md |
eval |
Execute JavaScript in the page | Live DOM access, complex transforms | eval.md |
eval --ref |
Execute JS scoped to a specific element | Element property extraction (text, attrs, styles) | ⚠️ Expression MUST be an arrow function: element => element.textContent |
extract, summarize, agent run |
AI-powered extraction | Natural language extraction (needs LLM key) | agent.md |
crawl |
Recursive crawling + bulk extraction | Multi-page traversal, seed-file processing | crawl.md |
swarm |
Parallel scraping across browser contexts | High-throughput extraction | swarm.md |
loop |
Repeated task execution with persistence | Monitoring, scheduled checks | loop.md |
state-save, state-load, cookie-*, *-storage-* |
Browser storage management | Auth state reuse, cookie manipulation | storage-state.md |
attach |
Connect to existing Chrome/Edge via CDP | Debug live browser, reuse auth — after attaching, check the printed actual browser; a ⚠ warning flags a channel mismatch (e.g. requested msedge but Chrome connected) | attach.md |
webdb export, webdb normalize |
Export cached pages, normalize URLs to database keys | Post-crawl content extraction, URL key lookup | webdb.md |
skills, skills get, skills path, skills unpack |
Bundled AI agent skill files | Refresh agent instructions, unpack skill files | skills.md |
skill-list, skill-info, skill-install, skill-uninstall, skill-reload |
Backend skill management | Install/manage server-side skills | skills.md |
screenshot, scroll, wait, resize |
Visual capture & viewport control | Screenshots, viewport sizing, scroll control | — |
tab-list, tab-new, tab-select, tab-close |
Tab management | Multi-tab workflows, session-scoped tab operations. See §Tab Management below. | — |
config |
Persistent CLI defaults (server, timeout, proxy, session) | Set default server URL, timeout, proxy, or session name. See §Configuration. | — |
Refreshing This Skill
The skills command retrieves bundled skill content that always matches the installed CLI version. Use it to get current instructions rather than relying on cached copies:
browser4-cli skills # List all bundled skills
browser4-cli skills get browser4-cli # Print this SKILL.md
browser4-cli skills get browser4-cli --full # Include all reference files
browser4-cli skills path # Print skills directory path
browser4-cli skills unpack # Unpack bundled skill files to disk
Set BROWSER4_SKILLS_DIR to override the skills directory location. Skill files are unpacked automatically during browser4-cli install (and refreshed by upgrade) into the versioned installation directory; unchanged files are skipped, so re-running is cheap. install / upgrade also copy the bundled skills into ~/.agents/skills so AI agents (e.g. Codex) can load them automatically (override with BROWSER4_AGENTS_SKILLS_DIR). Use skills unpack to refresh or relocate skill files without reinstalling.
4. Decision Trees
4a. Choosing an Extraction Method
📋 snapshot vs htmlsnapshot — the essential distinction:
snapshothtmlsnapshotWhat it captures Accessibility tree (AXTree) — semantic roles, names, refs Raw HTML DOM — full text content Primary use Interaction — get element refs for click, fill, type Extraction — get article text, data, attributes Output YAML tree with [ref=e5]handlesText/HTML/JSON via CSS selectors Key commands snapshot,snapshot grep,click <ref>htmlsnapshot get,query,inspectWhen to use "I need to click a button" or "find an input field" "I need to read the article text" or "extract prices" Rule of thumb: If you want to interact with elements →
snapshot. If you want to read content →htmlsnapshot.
⚠️ htmlsnapshot capture requirements — which commands need a prior capture:
Command Needs prior htmlsnapshotcapture?Notes htmlsnapshot(capture)— (this IS the capture) Stores the page's initial HTML for later extraction htmlsnapshot get/get allYes — requires stored snapshot Extracts text/html/attr via CSS selectors from the stored HTML htmlsnapshot inspectYes — requires stored snapshot Iterates CSS selectors from the stored HTML; returns "No HTML snapshot found" if missing htmlsnapshot summaryYes — requires stored snapshot Statistical summary of selectors on the stored page htmlsnapshot grepYes — requires stored snapshot Regex search over the stored HTML htmlsnapshot exportYes — requires stored snapshot Exports the stored HTML to a file htmlsnapshot queryNo — no capture needed Current page → queries the session's live DOM (seeded before the SQL runs; login/SPA state visible). Other URLs → independent webdb load, no session state If you get "No HTML snapshot found" or a timeout: either run
htmlsnapshotfirst to capture, or usehtmlsnapshot query— it needs no prior capture (current page reads the live DOM; an explicit URL is fetched independently).
⚠️ Important:
htmlsnapshotcaptures the current live DOM at capture time. Content added or modified by JavaScript before the capture (form submission results, dynamic updates, SPA route changes) is reflected — but only if you runhtmlsnapshot(capture) after the interaction. The stored snapshot becomes stale only if you do not re-capture after a navigation or interaction. For one-off live reads without a capture step, useeval. See §5 Critical Warnings for more.
Need to extract data from a page?
├─ Need to interact first (click, fill, scroll)?
│ → snapshot + refs, then re-capture htmlsnapshot after interacting, then extract
├─ Live-DOM single-field read WITHOUT a capture (post-interaction verify, quick read)?
│ → get text "<selector>" / get attr "<selector>" href (capture-free live read)
├─ Page has JS-updated content (after interaction, form submit, SPA)?
│ → eval --json for live DOM (use --stdin or --file on Windows)
├─ Static page, one field? → htmlsnapshot get text "<selector>"
├─ Static page, one field, ALL matches? → htmlsnapshot get all text "<selector>"
├─ Don't know the right CSS selector? → htmlsnapshot get text article (auto-discovers content)
├─ Static page, multiple correlated fields (title+price+url per item)?
│ → htmlsnapshot query with X-SQL DOM_LOAD_AND_SELECT
├─ Dynamic/complex JS logic needed? → eval --json
├─ Natural language ("find the product price")? → extract (needs LLM key)
└─ High volume, many pages? → crawl or swarm with --sql
4b. Choosing Bulk/Scale Approach
Need to process multiple pages?
├─ Single list page (products on one search results page)?
│ → htmlsnapshot query with DOM_LOAD_AND_SELECT
├─ Multiple known URLs (list in a file)? → crawl --seed-file urls.txt --depth 0 --sql @query.sql
├─ Crawl from a start URL (follow links)? → crawl <url> --out-link-selector "..." --depth N
├─ Need parallel execution (high throughput)? → swarm create → swarm query --seed-file ...
├─ Repeated monitoring (check every hour)? → loop -i 3600 -- eval "..."
└─ Just a few URLs in a shell script?
→ browser4-cli open --headless (once) then use goto for each URL; add wait between iterations
4c. Query Granularity: get vs get all vs query
| Command | Returns | Best for |
|---|---|---|
htmlsnapshot get text ".price" |
First match only (string) | Single value, quick check |
htmlsnapshot get all text ".price" |
All matches (JSON array) | Validate a selector returns expected count |
htmlsnapshot query --sql "SELECT ..." |
Correlated multi-field rows | Title + price + URL per product card |
Warning: Multiple get all calls produce unaligned arrays (different lengths, different order). For correlated fields, use query with DOM_LOAD_AND_SELECT scoped to a parent container.
4d. Structuring Extracted Pages (WebMiner)
WebMiner runs ML clustering on downloaded HTML files to produce structured spreadsheets and interactive reports — no LLM tokens, everything runs locally.
webminer is a first-class Browser4 CLI citizen — browser4-cli webminer <command> installs and runs the tool without PowerShell:
Have HTML files and want structured data — without tokens?
├─ < 1,000 pages (small to medium)? → WebMiner Free (SMILE ML engine)
│ browser4-cli webminer install
│ browser4-cli webminer all ./html-pages/
│ → Interactive HTML report + Excel spreadsheets — everything local, zero cost
├─ > 1,000 pages (production scale)? → WebMiner Commercial (Apache Spark ML)
│ Same encode → cluster → views pipeline, distributed across machines
│ → Scales to 100K+ pages/day
└─ Need to acquire pages first?
├─ Single pages: browser4-cli open --headless → htmlsnapshot → htmlsnapshot export
├─ Bulk download: browser4-cli crawl --seed-file urls.txt --depth 0 --refresh
└─ High throughput: browser4-cli swarm create → swarm query --seed-file ...
Then feed the HTML directory to WebMiner
Swarm acquisition → on-disk corpus:
swarm queryreturns structured rows, not files. Every page the swarm fetched is cached in the backend page store (webdb), so stage the corpus withbrowser4-cli webdb export "url1,url2,…" <output-dir>(URLs comma-separated; see webdb.md) and pointwebminer allat that directory. For on-disk HTML as the primary goal, prefercrawl --depth 0— swarm shines when extraction throughput matters more than the files.
Pipeline: encode (HTML → feature vectors → CSV) → cluster (KMeans, auto-detected K) → views (interactive HTML report + Excel spreadsheets)
Clustering granularity: the ML engine clusters DOM node rows, not whole pages —
kmeans-result/*/predictionAndOriginalFeatures/result.csvrows are nodes, with each row's page attribution in the far-righturlcolumn, and the auto-detected K scales with node count (dozens of clusters for a small corpus). Use the generated views (index.html/*.xlsx) for page-level analysis; treatresult.csvas the node-level detail table.
Free tier (SMILE): Single-machine ML via the SMILE library. Handles small-to-medium datasets (< 1,000 pages). Ideal for ad-hoc analysis, prototyping, and one-off extraction tasks.
Commercial tier (Apache Spark ML): Distributed clustering for production workloads. Scales to 100K+ pages/day. Same pipeline, enterprise throughput.
CLI usage (no backend, no PowerShell needed):
| Command | Purpose |
|---|---|
webminer |
Show installed version, Java 17+ status, and subcommand list |
webminer install [version] |
Download + verify scent-miner.jar (GitHub → OSS mirror), install to ~/.scent/webminer |
webminer update |
Update to the latest release |
webminer version |
Show installed and latest versions |
webminer uninstall |
Remove the installed release |
webminer run-example |
Download the sample dataset and run the full pipeline (needs 7-Zip) |
webminer all <html-dir> |
Full pipeline: encode → cluster → views (--max-files, --output, --resume) |
webminer views <result-dir> |
Rebuild the interactive views from an existing run |
Requires JDK 17+ (auto-detected from JAVA_HOME, common paths, or PATH). Any other command is forwarded verbatim to scent-miner.jar (e.g. webminer encode <dir>).
Install:
browser4-cli webminer install(or the legacy launcher.\webminer.ps1 installfrom the web-miner project). The JAR is also downloadable from web-miner releases.
See web-miner/SKILL.md for the full reference.
4e. X-SQL Quickstart Template
X-SQL lets you extract correlated fields (e.g., title + price + URL) from a list page using a scoped CSS selector and standard SQL. Copy this template, swap the selectors and column names, and you have a working query:
SELECT
DOM_FIRST_TEXT(DOM, 'h2') AS title,
DOM_FIRST_TEXT(DOM, '.price') AS price,
DOM_BASE_URI(DOM) AS url
FROM
DOM_LOAD_AND_SELECT(@url, '.product-card')
Save to a file (avoids shell quoting issues):
# 1. Write the query (copy and customize)
cat > query.sql << 'XSQL'
SELECT
DOM_FIRST_TEXT(DOM, 'h2') AS title,
DOM_FIRST_TEXT(DOM, '.price') AS price,
DOM_BASE_URI(DOM) AS url
FROM
DOM_LOAD_AND_SELECT(@url, '.product-card')
XSQL
# 2. Discover the right CSS selector to replace .product-card:
browser4-cli htmlsnapshot inspect --selector-base64 <base64-of-selector>
# 3. Run it
browser4-cli htmlsnapshot query "https://example.com/products" --sql @query.sql
# Default output is the raw JSON response envelope (machine-readable).
# For human-readable output add --format table (csv also available;
# --result-only prints just the resultSet):
browser4-cli htmlsnapshot query "https://example.com/products" --sql @query.sql --format table
Exit codes: htmlsnapshot query exits 0 when the response envelope
reports success — a 200 with an empty resultSet counts as success
("no rows matched", not an error). It exits nonzero when the server returns
an error envelope (417 Expectation Failed — the scrape session closed
before the query ran — or a 5xx with an empty resultSet), so scripts can
detect failure without parsing the JSON.
Critical syntax rules (H2 SQL engine — violating these produces opaque errors):
| Rule | Correct | Wrong |
|---|---|---|
| CSS selectors use single quotes (SQL string literals) | 'h2', '.price' |
"h2" (SQL identifier) |
@url placeholder is unquoted |
@url |
'@url' (literal string) |
FROM source is always DOM_LOAD_AND_SELECT |
DOM_LOAD_AND_SELECT(@url, '...') |
Any other table name |
No CTEs (WITH), no JOIN, no subqueries |
Simple SELECT … FROM … |
WITH t AS (…) SELECT … |
Discover selectors before writing the query:
browser4-cli htmlsnapshot inspect # recurring-pattern discovery (list/grid pages)
browser4-cli htmlsnapshot summary # visual clustering (detail/single-block pages)
browser4-cli htmlsnapshot get text ".price" --all # quick test: does this selector match elements?
htmlsnapshot inspect finds recurring patterns — it is built for list/grid pages (search results, product cards, tables). A single product/article/detail page has no repeating block, so inspect may surface nothing or an unrelated side rail; when that happens it prints "No recurring pattern found". For detail pages use htmlsnapshot summary (visual clustering) or htmlsnapshot get with explicit selectors (htmlsnapshot get text "h1", get attr "#product-image" src).
Common mistakes and solutions:
| Symptom | Likely cause | Fix |
|---|---|---|
Column "h2" not found |
Double quotes around CSS selector → treated as SQL column name | Use single quotes: 'h2' |
Table "..." not found |
Wrong FROM source or quoted @url |
Use DOM_LOAD_AND_SELECT(@url, 'selector') |
Hexadecimal string contains non-hex character (417) |
DOM_FIRST_FLOAT/DOM_FIRST_INTEGER compared to a numeric literal in WHERE — the function returns a custom H2 value type that predicates cannot compare (works in SELECT/ORDER BY) |
Wrap in a numeric cast: WHERE CAST(DOM_FIRST_FLOAT(DOM, '.price', 0.0) AS DOUBLE) >= 25.0 (or STR_FIRST_FLOAT(DOM_FIRST_TEXT(DOM, '.price'), 0.0)) — always pass the default argument explicitly (the registered form is DOM_FIRST_FLOAT(DOM, sel, default); the 2-argument shorthand is not portable across engine builds), see x-sql.md |
| Empty result set | Selector doesn't match any elements | Run htmlsnapshot inspect to find valid selectors |
| Empty result set (no error) | Filtering images with a PowerCSS :expr(...) selector passed to a DOM_*_IMG function — the img-scanning path ignores :expr and matches nothing |
Use an attribute path that honors :expr: DOM_FIRST_ATTR(DOM, 'img:expr(src^=https://cdn)', 'src') or DOM_SELECT_FIRST(DOM, sel) + DOM_ABS_SRC — see x-sql-dom-select-functions.md |
DOM_FIRST_HREF(DOM, '.link') returns '' silently |
The href-scanning helper needs a tag context; class-only selectors can resolve to an empty string with exit 0 (indistinguishable from "no link") | Use the tag-qualified form DOM_FIRST_HREF(DOM, 'a.link'), or DOM_FIRST_ATTR(DOM, sel, 'href') which accepts any selector — see x-sql.md |
Syntax error in SQL statement |
--sql value contains shell-escaped characters |
Use --sql @query.sql instead of inline SQL |
5. Critical Warnings
Warning: Refs are effectively single-use. Re-snapshot after any interaction before using refs again, and always do so after
goto,reload, and tab switches. On reactive pages, even form commands can leave earlier refs stale. Never store refs across navigations or assume a pre-interaction ref is still valid.
Warning: CSS selectors are tied to live websites — they break when sites change their HTML. Always discover selectors with
htmlsnapshot inspectorhtmlsnapshot summarybefore extraction. Treat scenario examples as patterns, not copy-paste recipes.
Warning: Shell quoting on Windows — complex JS/SQL with nested quotes causes escaping issues. Prefer
--sql @file.sql(read from file),--sql-stdin(piped),--sql-base64(encoded), oreval --file/eval --stdin/eval --base64(JS from file or base64). Forhtmlsnapshot inspect, use@file,--stdin, or--selector-base64. Never inline--sql "..."with double-quoted CSS selectors on Windows. On PowerShell, always quote@filepaths (--sql "@query.sql") — an unquoted@is read as the splatting operator. See shell-quoting.md for the full workaround workflow.Tip: To generate base64 for
eval --base64:echo -n 'document.title' | base64(Linux/macOS) or[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes('document.title'))(PowerShell).⚠️ Important — eval with
--ref: When scoping evaluation to an element with--ref(or positional[ref]), the expression MUST be an arrow function:element => element.textContent. The DOM element is passed as the first argument. Writingelement.textContentorthis.textContentwill returnnull— this is the #1 user mistake with element-scoped eval.
Warning: Don't cat snapshot files — they can exceed 256KB. The same applies to
--stdout, which may dump large accessibility trees (63KB+ for content-rich pages). Use viewport pagination (snapshot -v 0),snapshot grep <pattern>, orsnapshot --stdout --page 1instead. For targeted extraction, prefersnapshot greporhtmlsnapshotcommands over full-tree dumps.
Note: Output pagination defaults —
get html,get all html, andgreppaginate at 2K lines.get textandget all textare not paginated by default. Use--allto disable pagination, or--page Nfor subsequent pages.
Note —
snapshot --stdout/--rawpagination: large stdout trees are paginated at 2000 lines/page by default. When truncated, the footer goes to stderr, and when stdout is a pipe/redirect a# … output truncated: showing N of M lines — re-run with --all (or --page-size 0) for the full tree.hint is appended to stdout so the cut is visible in captured output.--page-size 0(or--all) disables paging. For very large pages, prefer bounding the capture with-v N,--depth,--selector, or--no-boxesover full-tree dumps.
Snapshot modes — when to use
-v 0vs-ivs default:
Mode What it shows Best for snapshot(default)Full AX tree with all element refs General exploration, first look at a page snapshot -v 0Current visible screen (a single screen-height viewport chunk) Long pages — read one chunk at a time to keep output small. Use -v allfor the entire pagesnapshot -iInteractive-oriented layout — inner text is aggregated into the enclosing element's name so each ref line reads as a self-contained target. Not a strict interactive-only filter: addressable headings, paragraphs and generic containers remain Quick orientation before acting via refs; form-heavy pages where most lines are controls anyway. To bound size use -v 0viewport pagination,--selector, orhtmlsnapshothtmlsnapshotStatic HTML (CSS selectors) Content extraction (text, attributes), when you need CSS selectors instead of AX refs
-idoes not shrink the tree: the interactive pass aggregates text into element names; it does not strip non-interactive containers (addressable headings,<div>wrappers etc. remain). Pair-iwith-v 0(snapshot -i -v 0) for one focused screenful, and use `htmls
…(truncated)
