thedixitjain Crawlberg
Crawlberg is a Rust-native web crawler and scraper. It fetches static HTML
with reqwest, falls back to headless Chrome when a page needs JS or trips a
WAF, and converts every result to clean Markdown via the built-in
HTML→Markdown engine.
Use this skill when the user wants to:
- Scrape a single URL to Markdown plus structured metadata.
- Crawl a site following links bounded by depth, page count, and concurrency.
- Enumerate URLs from sitemaps without paying for rendering.
- Drive a real browser (click, type, scroll) and capture the resulting DOM.
- Run the same operations from another agent harness via MCP tools.
Installation
The plugin shells out to a crawlberg binary on PATH. Install one of:
brew install xberg-io/tap/crawlberg
# or run without a persistent install (the CLI proxy package self-installs the binary):
npx @xberg-io/crawlberg-cli --help
uvx --from crawlberg-cli crawlberg --help
# or build from source:
cargo install crawlberg-cli --features all
The serve and mcp subcommands are gated behind non-default cargo features
(api and mcp). The Homebrew tap is built with all features, so both
subcommands work out of the box. A from-source build must pass
--features mcp (and --features api for serve), or --features all, to
include them.
Verify:
crawlberg --version
Headless fallback needs Chrome/Chromium reachable locally (chromiumoxide
launches it on demand). Skip the install if you only plan to use
--browser-mode never.
Command map
crawlberg scrape <url> # single page → JSON or Markdown
crawlberg crawl <url...> # follow links, BFS, depth-bounded
crawlberg map <url> # enumerate URLs via sitemaps + link extraction
crawlberg interact <url> # browser actions: click, type, scroll
crawlberg batch-scrape <url...> # scrape many URLs concurrently
crawlberg batch-crawl <url...> # crawl many seed URLs concurrently
crawlberg download <url> # download a document, report file metadata
crawlberg citations <input> # markdown links → numbered citations (text or @file.md)
crawlberg version # print the crawlberg version as JSON
crawlberg mcp # MCP server (stdio) — auto-registered (`mcp` feature)
crawlberg serve # REST API server (`api` feature)
crawl also handles batching implicitly: pass multiple seed URLs and it fans
out via batch_crawl internally. The explicit batch-scrape and batch-crawl
subcommands expose the same concurrency for many independent URLs.
Per-subcommand flags:
| Subcommand | Positional | Key flags |
|---|---|---|
scrape |
<url> |
--proxy, --user-agent (plus shared flags below) |
crawl |
<url...> |
--depth/-d (2), --max-pages/-n, --concurrent/-c (10), --rate-limit (200), --stay-on-domain, --proxy, --user-agent |
map |
<url> |
--limit, --search |
interact |
<url> |
--actions <json> (required) |
batch-scrape |
<url...> |
--concurrent/-c (10), --proxy, --user-agent |
batch-crawl |
<url...> |
--depth/-d (2), --max-pages/-n, --concurrent/-c (10), --rate-limit (200), --stay-on-domain, --proxy, --user-agent |
download |
<url> |
--max-size |
citations |
<input> |
none (input is markdown text or @file.md) |
version |
— | none |
serve |
— | --host (0.0.0.0), --port (3000) |
mcp |
— | none (stdio transport) |
Shared flags
| Flag | Default | Notes |
|---|---|---|
--format |
json |
json or markdown. |
--timeout |
30000 |
Request timeout in milliseconds. |
--browser-mode |
auto |
auto, always, or never. |
--browser-endpoint |
— | Optional CDP ws:// or wss:// URL. |
--respect-robots-txt |
off | Pass to obey robots.txt. |
--config <json> |
— | Inline JSON or @file.json to override defaults. |
The --config flag accepts the full CrawlConfig schema. Anything you set
explicitly on the CLI overrides the corresponding JSON field.
These shared flags apply to the crawl/scrape-family subcommands. --format,
--browser-mode, --browser-endpoint, and --config cover scrape, crawl,
map, interact, batch-scrape, and batch-crawl; download takes
--timeout, --browser-mode, --browser-endpoint, --max-size, and
--config (no --format). --respect-robots-txt applies to scrape,
crawl, map, batch-scrape, and batch-crawl. citations and version
take no shared flags.
Scrape a single page
crawlberg scrape https://example.com --format markdown
JSON output (default) carries the rendered Markdown, page metadata
(PageMetadata), links by category, images, feeds, JSON-LD blocks, and
HTTP response metadata. Use Markdown output when piping into a file the user
will read.
See the scraping-html-to-markdown skill for the full flag surface.
Crawl a site
crawlberg crawl https://example.com \
--depth 3 --max-pages 200 --concurrent 8 --rate-limit 250 \
--stay-on-domain --respect-robots-txt --format markdown
Crawling is BFS by default, bounded by --depth, --max-pages, and
--concurrent. Per-domain politeness is enforced by --rate-limit
(milliseconds between requests to the same origin).
See the crawling-a-site skill for the recommended defaults and the full
flag surface.
Map URLs
crawlberg map https://example.com --limit 500 --search docs --format markdown
map reads sitemap.xml (and nested sitemaps), then falls back to link
extraction from the seed page. It does not render pages — use it to plan a
crawl or to feed URLs into another tool.
Browser interaction
crawlberg interact https://example.com \
--actions '[{"type":"click","selector":"#load-more"},
{"type":"wait","milliseconds":500},
{"type":"scrape"}]'
Action types are click, type, press, scroll, wait, screenshot,
executeJs, and scrape (to wait for an element, use wait with a
selector field). The result wraps the final HTML under
interaction.final_html. See the automating-the-browser skill for the full
action schema and limits.
MCP server
When this plugin is installed in a Claude Code / Codex / Cursor / Gemini / opencode harness, the MCP server is auto-registered:
crawlberg mcp
mcp is a stdio-transport server and takes no arguments. It requires a binary
built with the mcp feature (see Installation).
The server registers nine tools (the same set is served over the Streamable
HTTP transport when running crawlberg serve):
| Tool | Purpose | Parameters |
|---|---|---|
scrape |
Scrape one URL to Markdown or JSON (content, metadata, links). | url (required), format (markdown|json), use_browser (bool — force browser) |
crawl |
Follow links from a URL, bounded by depth/page count. | url (required), max_depth, max_pages, format, stay_on_domain |
map |
Discover all URLs via links and sitemaps. | url (required), limit, search, respect_robots_txt |
batch_scrape |
Scrape multiple URLs concurrently. | urls (required array), format, concurrency |
batch_crawl |
Crawl multiple seed URLs concurrently. | urls (required array), max_depth, max_pages, format, stay_on_domain, concurrency |
download |
Download a document and return file metadata. | url (required), max_size |
interact |
Execute browser actions on a page (mutating/destructive). | url (required), actions (required array of action objects) |
generate_citations |
Rewrite markdown links as numbered citations + reference list. | markdown (required) |
get_version |
Return the crawlberg library version. | none |
Prefer MCP tools over shelling out when both are available:
- Typed schemas surface argument errors before the call.
- Results stream back as structured tool output instead of stdout text.
- No
--formatjuggling — the harness pulls whatever shape it needs.
Fall back to the CLI when you need to script a pipeline, capture stderr, or chain with shell tools.
Headless fallback
In --browser-mode auto (default), the engine:
- Fetches statically via
reqwest. - Detects WAF blocks (8 vendors) and JS-only shells.
- Re-fetches through headless Chrome with a real fingerprint when needed.
Force the browser path with --browser-mode always when you already know
the page needs JS. Use --browser-mode never for hot loops where the cost
of a stray Chrome launch is unacceptable.
Point --browser-endpoint ws://host:9222/devtools/browser/<id> at an
already-running Chrome to skip the local launch.
See the headless-fallback skill for symptoms, costs, and external-CDP
patterns.
Output formats
| Mode | Use when |
|---|---|
json |
Downstream consumer needs metadata, links, images, etc. |
markdown |
Human reader or LLM-context payload. |
Markdown output skips metadata. If you need both, run with --format json
and read result.markdown.content.
Robots, rate limits, ethics
--respect-robots-txtis off by default; pass it for any crawl on a host you do not own.- The default
--rate-limit 200already produces a polite cadence; raise it for shared hosts. - Identify the crawler honestly via
--user-agent. Do not impersonate a browser unless the operator has approved it.
Cross-references
skills/crawling-a-site/SKILL.md— multi-page crawl with depth, page caps, concurrency, rate limits, and domain scoping.skills/scraping-html-to-markdown/SKILL.md— single-page rendering, the Markdown output shape, and common pitfalls.skills/mapping-urls/SKILL.md—map: sitemap + link URL discovery, filtering, and seeding a crawl.skills/automating-the-browser/SKILL.md—interact: the full scripted action schema, limits, and result shape.skills/serving-the-api/SKILL.md—serve: the Firecrawl-v1-compatible REST API server and its endpoints.skills/headless-fallback/SKILL.md— when and how to force the browser backend.
Source: hashgraph-online/awesome-codex-plugins → plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawlberg/SKILL.md