0xDarkMatter Flarecrawl Operations
Cloudflare Browser Rendering CLI — Firecrawl-compatible, cost-efficient at scale. Renders JavaScript, extracts markdown/HTML, crawls sites, takes screenshots, generates PDFs, AI-extracts structured data, fingerprints tech stacks, and cracks anti-bot walls. Also ships an MCP server and a CDP mode for persistent, authenticated browser sessions.
Orient first.
flarecrawl guideprints the packaged agent reference (when/why each command, JSON shapes, exit codes, footguns).flarecrawl guide <topic>drills in (aliases:hard-targets,json,errors,rules,auth).flarecrawl guide --listshows topics. Treatguideas the source of truth; this skill is the quick map.
Auth Check
flarecrawl auth status --json
# Exit code 2 = not authenticated → run: flarecrawl auth login
flarecrawl usage --json # free tier: 10 min/day (600,000ms) — check before big jobs
MCP Server (for agents)
Flarecrawl exposes a Model Context Protocol stdio server — 36 tools in a three-tier surface — so an agent can call the toolkit directly instead of shelling out.
uv pip install 'flarecrawl[mcp]'
flarecrawl mcp # start stdio server
flarecrawl mcp --read-only # excludes page_interact, site_download, p6_raw, recipe_run_raw, spider_raw
Wire into .mcp.json:
{"mcpServers": {"flarecrawl": {"command": "flarecrawl", "args": ["mcp"]}}}
| Tier | Count | When | Examples |
|---|---|---|---|
| Orientation | 5 | session start / capability check | capabilities(), guide(), diagnostics() |
| T1 Composite | 5 | ~80% of tasks | read_page(), research_web(), site_overview() |
| T2 Curated | 17 | single-resource ops | web_search(), tech_detect(), crawl_start() |
| T3 Raw | 9 | full CLI fidelity | scrape_raw(), p6_raw(), spider_raw() |
Always call capabilities() first — one call returns the full catalogue,
permissions, coverage gaps, and worked recipes. Content tools default to
--agent-safe. Errors return {"ok": false, "error": {"code", "next_steps"}};
meta.blocked verdicts auto-suggest recovery (Akamai → stealth/p6; CF-1020 →
terminal). Coverage gaps (CLI only): videos, authcrawl, frontier, batch,
auth login/logout, cache clear, rules *, cdp *, webmcp *, some
session subcommands, --interactive/--headed.
Core Operations
# Scrape → markdown (or html/links/images/summary/schema)
flarecrawl scrape https://example.com --json
flarecrawl scrape URL --format html --json
flarecrawl scrape URL --js # SPAs: wait for networkidle0
flarecrawl scrape URL --only-main-content # strip nav/footer (link-density gated)
# Crawl a site (async; --wait to block)
flarecrawl crawl https://docs.example.com --wait --limit 50 --json
flarecrawl crawl JOB_ID --ndjson --fields url,markdown # stream large results
# Discover URLs
flarecrawl map URL --json # links on one page
flarecrawl discover URL --json # sitemaps + RSS/Atom + links
flarecrawl spider URL --limit 500 # direct HTTP crawl, NO browser cost, resumable
# Fetch by content type (4-branch routing: binary / JSON / raw-text / HTML→CF)
flarecrawl fetch URL --json # XML/CSV/RSS/KML/YAML returned verbatim, no CF auth
flarecrawl fetch URL -o file.pdf # binary → file
# AI extraction (Workers AI, $0) — see flarecrawl-extraction skill
flarecrawl extract "Get product names and prices" --urls URL --json
# Save site / media
flarecrawl download URL --limit 50 # → .flarecrawl/<domain>/<page>.md
flarecrawl screenshot URL -o page.png --full-page
flarecrawl pdf URL -o page.pdf
flarecrawl favicon URL --all --json
Tech Detection (Wappalyzer)
flarecrawl tech-detect URL --json # CMS, framework, CDN, analytics, SaaS — single GET, 0 CF time
flarecrawl tech-detect URL --json --exclude-categories "Miscellaneous,Security,Tag managers,RUM" # clean stack picture
flarecrawl tech-detect -i urls.txt --workers 10 --json # batch
flarecrawl tech-detect URL --render --json # Playwright + JS-globals probe (~880 more fingerprints)
flarecrawl scrape URL --tech-detect --json # inline on scrape/crawl/fetch too
Hard Targets & Anti-Bot
For sites that fingerprint TLS, return stubs to CF, or hide data in JS state:
# Enhanced extraction + TLS impersonation (needs curl_cffi)
flarecrawl scrape URL --paywall --stealth --json # metadata.source shows the strategy that won
# Local Chromium backend (bypass CF stub) + capture XHR bodies
flarecrawl scrape URL --js --browser local --headed
flarecrawl scrape URL --js --capture-pattern "*.csv,*.json" --capture-dir ./out/
# Mass-download from a captured manifest, reusing session + stealth TLS
flarecrawl scrape URL --then-fetch-from manifest.csv --then-fetch-column "Link" --then-fetch-output ./files/
# P6: mint→replay (local Chromium mints cookie shells → curl_cffi replays w/ real JA3/JA4)
flarecrawl p6 https://site/ --jar jar.json --target https://site/api --output-dir ./out
flarecrawl session inspect @site # offline jar freshness; exit ≠0 unless fresh
# YAML multi-step browser flows (resumable)
flarecrawl recipe flow.yml --dry-run
flarecrawl recipe flow.yml --resume
meta.blocked {blocked, vendor, kind, terminal, signal} is the bot-wall source
of truth (scrape CDP, fetch --json, recipe). terminal: true (CF-1020) =
non-bypassable, don't burn a re-mint. Don't string-match block pages yourself.
CDP — Persistent / Authenticated Sessions
uv pip install websockets # or flarecrawl[cdp]
flarecrawl scrape URL --interactive --json # log in via DevTools (OAuth/2FA/CAPTCHA), cookies auto-saved
flarecrawl scrape URL --cdp --keep-alive 60 --save-cookies session.json --har traffic.har --json
flarecrawl scrape URL --load-cookies session.json --cdp --json
flarecrawl cdp sessions --json && flarecrawl cdp close
CDP flags (--interactive, --live-view, --record, --save/load-cookies,
--keep-alive, --browser-cookies) all auto-promote to --cdp.
Agent-Safe Mode
flarecrawl scrape URL --agent-safe --json # sanitise adversarial content (DeepMind "AI Agent Traps")
13 sanitisers, two phases (HTML + text). metadata.agentSafety reports findings.
Prompt injection removed (short-line bias <200 chars); semantic manipulation
flagged, not removed. Works on scrape/crawl/download/extract and --stdin.
Batch & Parallel
scrape and extract support --batch/-b and --workers/-w:
flarecrawl scrape --batch urls.txt --workers 5
flarecrawl map URL --json | jq -r '.data[]' > urls.txt && flarecrawl scrape --batch urls.txt --workers 5
- Input: plain text (one/line,
#comments), JSON array, or NDJSON (auto-detected) - Output: NDJSON
{index, status, data/error}, sorted by index - Workers: default 3, max 50 (CF raised concurrency to 120). Override with
FLARECRAWL_MAX_WORKERS - Partial failures don't stop processing; batch fails fast on auth/permission errors
Output Interpretation
{"data": {"url": "...", "content": "# Markdown...", "elapsed": 2.9}, "meta": {"format": "markdown"}}
| Field | Where | Description |
|---|---|---|
data.content |
scrape | Markdown or HTML |
data.elapsed |
scrape | Seconds to fetch |
data.records[] |
crawl | Crawled pages (.url, .markdown, .status) |
data.job_id / data.total / data.browser_seconds |
crawl | Job id, page count, browser time |
metadata.source |
scrape | content-negotiation (0 browser time) vs browser-rendering |
meta.blocked |
scrape/fetch/recipe | Bot-wall verdict |
Gotchas
- Free tier: 10 min/day, 3 concurrent browsers →
--workers 3. Paid: up to--workers 50 - Crawl is async — use
--wait(or poll--status); jobs persist 14 days on CF - Markdown content negotiation is automatic —
scrapetriesAccept: text/markdownfirst (0 browser time);--no-negotiateto force browser --inputis legacy — use--batch- Binary outputs (screenshot/pdf) save to files;
--jsonfor base64 - Auto-retry on 429/502/503 with backoff (3 attempts)
--cdp,--stealth,--paywall,--browser local, recipes need optional extras (websockets / curl_cffi / playwright / PyYAML)- Exit codes: 0 ok · 1 error · 2 auth · 3 not-found · 4 validation · 5 forbidden · 7 rate-limited
Pipe Patterns
| Chain | Command |
|---|---|
| → content | flarecrawl scrape URL --json | jq '.data.content' |
| → map then scrape | flarecrawl map URL --json | jq -r '.data[]' > urls.txt && flarecrawl scrape --batch urls.txt |
| → filter batch | flarecrawl scrape --batch urls.txt | jq 'select(.status=="ok") | .data.content' |
| → stream crawl | flarecrawl crawl JOB_ID --ndjson --fields url,markdown | jq -r '.markdown' |
| → tech-detect feed | flarecrawl tech-detect -i urls.txt --workers 10 --json | jq -r '.data[].technologies[].name' |
For full flag reference and JSON shapes, run flarecrawl guide <topic> or read the repo AGENTS.md.