Mass Scraping — Crawl4AI + Camoufox
Verified 2026-08-27 (macOS arm64). Both engines and all three templates smoke-tested.
Install (if crawl4ai-python is not on PATH)
- crawl4ai —
github.com/unclecode/crawl4ai · PyPI crawl4ai
- camoufox —
github.com/daijro/camoufox · PyPI camoufox[geoip]
- curl_cffi —
github.com/lexiforest/curl_cffi · PyPI curl_cffi
Dedicated venv, then crawl4ai-setup and python -m camoufox fetch. Four traps, each verified here:
- Python 3.10–3.13 only. 3.14 is not supported — if
python3 is 3.14, install 3.13 explicitly.
crawl4ai, not crawl4ai[all]. The extra pulls torch + transformers (GBs) that scraping never uses.crawl4ai-setup, not playwright install chromium. Setup also installs Patchright, which
rung 3 needs; without it the undetected adapter dies.- Shim the venv python with a wrapper, never a symlink. A symlinked venv python resolves
sys.prefix to the system interpreter and cannot import crawl4ai:printf '#!/bin/sh\nexec "$VENV/bin/python" "$@"\n' > ~/.local/bin/crawl4ai-python
Entry points
crwl https://example.com -o md -bc # one page → markdown
crawl4ai-python script.py # anything bigger
Both are shims into the dedicated venv. System python3 cannot import crawl4ai/camoufox — always
crawl4ai-python or the venv binary directly. Don't pip-install these into another environment.
crwl: -o md · -o all (JSON) · -bc (bypass cache) · --deep-crawl bfs --max-pages N.
-o json fails without an extraction schema — use -o all.
Escalation ladder — climb only on failure
| # |
Approach |
Cost |
When |
| 0 |
curl_cffi (TLS/JA3 impersonation, no browser) |
~10ms/page |
Try first. Clears more than expected — see 03 |
| 1 |
crwl / plain AsyncWebCrawler |
~1s/page |
Page needs JS |
| 2 |
BrowserConfig(enable_stealth=True) |
~0 |
Rung 1 → challenge/403 |
| 3 |
UndetectedAdapter() (= Patchright) |
+~1s/page |
Rung 2 still blocked |
| 4 |
Camoufox + raw:// bridge |
200MB/instance, slow |
Chromium fingerprint-blocked |
| 5 |
Residential proxies |
$ |
Blocked by IP, not fingerprint |
Rungs 2 and 3 are mutually exclusive — browser_manager.py:763:
if self.config.enable_stealth and not self.use_undetected:. The undetected adapter silently drops
enable_stealth. "Combine both for max evasion" advice is false; verified in installed source.
Captcha
reCAPTCHA v3 and Turnstile are not solvable offline — behavioral/TLS scoring, no image. The only
play is not triggering them (rungs 2-4). Repos named *-turnstile-solver = token harvesters
(minutes-long TTL) or paid-API wrappers.
Simple image captchas are offline-solvable → ddddocr, not installed. Ask first.
Still blocked at rung 5? Say so. Never return partial data silently.
References
| Need |
File |
arun_many, dispatchers, deep crawl, extraction, tuning |
references/01-crawl4ai-scale.md |
Fingerprints, proxy+geoip, humanize, raw:// bridge |
references/02-camoufox.md |
| Benchmarks, failure signatures, what beats what |
references/03-anti-bot-reality.md |
templates/ — copy, don't retype: mass_crawl.py (rungs 1-3) · camoufox_bridge.py (rung 4) ·
deep_crawl.py (whole site).
Non-negotiables
cache_mode=CacheMode.BYPASS / -bc — default ENABLED serves stale pages and still reports success.- One
AsyncWebCrawler context for all URLs. Per-URL async with relaunches the browser (~50x slower).
- Check
result.success per item — arun_many returns failures, never raises.
- Respect robots.txt + rate limits unless the target is the user's. Use
RateLimiter, not a bare loop.
1---2name: web-scraping3description: Mass crawl/scrape sites incl. anti-bot (Cloudflare, Akamai, DataDome, PerimeterX). Crawl4AI = scale (arun_many, dispatchers, deep crawl, LLM-ready markdown); Camoufox = fingerprint evasion. For: scrape N pages, crawl a site, extract structured data, 403/challenge/bot-detection, captcha.4---56# Mass Scraping — Crawl4AI + Camoufox78Verified 2026-08-27 (macOS arm64). Both engines and all three templates smoke-tested.910## Install (if `crawl4ai-python` is not on PATH)1112- **crawl4ai** — `github.com/unclecode/crawl4ai` · PyPI `crawl4ai`13- **camoufox** — `github.com/daijro/camoufox` · PyPI `camoufox[geoip]`14- **curl_cffi** — `github.com/lexiforest/curl_cffi` · PyPI `curl_cffi`1516Dedicated venv, then `crawl4ai-setup` and `python -m camoufox fetch`. Four traps, each verified here:17181. **Python 3.10–3.13 only.** 3.14 is not supported — if `python3` is 3.14, install 3.13 explicitly.192. **`crawl4ai`, not `crawl4ai[all]`.** The extra pulls torch + transformers (GBs) that scraping never uses.203. **`crawl4ai-setup`, not `playwright install chromium`.** Setup also installs **Patchright**, which21 rung 3 needs; without it the undetected adapter dies.224. **Shim the venv python with a wrapper, never a symlink.** A symlinked venv python resolves23 `sys.prefix` to the system interpreter and cannot import crawl4ai:24 ```sh25 printf '#!/bin/sh\nexec "$VENV/bin/python" "$@"\n' > ~/.local/bin/crawl4ai-python26 ```2728## Entry points2930```bash31crwl https://example.com -o md -bc # one page → markdown32crawl4ai-python script.py # anything bigger33```3435Both are shims into the dedicated venv. **System `python3` cannot import crawl4ai/camoufox** — always36`crawl4ai-python` or the venv binary directly. Don't pip-install these into another environment.3738`crwl`: `-o md` · `-o all` (JSON) · `-bc` (bypass cache) · `--deep-crawl bfs --max-pages N`.39`-o json` fails without an extraction schema — use `-o all`.4041## Escalation ladder — climb only on failure4243| # | Approach | Cost | When |44|---|---|---|---|45| 0 | `curl_cffi` (TLS/JA3 impersonation, no browser) | ~10ms/page | **Try first.** Clears more than expected — see `03` |46| 1 | `crwl` / plain `AsyncWebCrawler` | ~1s/page | Page needs JS |47| 2 | `BrowserConfig(enable_stealth=True)` | ~0 | Rung 1 → challenge/403 |48| 3 | `UndetectedAdapter()` (= Patchright) | +~1s/page | Rung 2 still blocked |49| 4 | Camoufox + `raw://` bridge | 200MB/instance, slow | Chromium fingerprint-blocked |50| 5 | Residential proxies | $ | Blocked by IP, not fingerprint |5152**Rungs 2 and 3 are mutually exclusive** — `browser_manager.py:763`:53`if self.config.enable_stealth and not self.use_undetected:`. The undetected adapter silently drops54`enable_stealth`. "Combine both for max evasion" advice is false; verified in installed source.5556## Captcha5758**reCAPTCHA v3 and Turnstile are not solvable offline** — behavioral/TLS scoring, no image. The only59play is not triggering them (rungs 2-4). Repos named `*-turnstile-solver` = token harvesters60(minutes-long TTL) or paid-API wrappers.6162Simple image captchas *are* offline-solvable → `ddddocr`, not installed. Ask first.6364Still blocked at rung 5? Say so. Never return partial data silently.6566## References6768| Need | File |69|---|---|70| `arun_many`, dispatchers, deep crawl, extraction, tuning | `references/01-crawl4ai-scale.md` |71| Fingerprints, proxy+geoip, humanize, `raw://` bridge | `references/02-camoufox.md` |72| Benchmarks, failure signatures, what beats what | `references/03-anti-bot-reality.md` |7374`templates/` — copy, don't retype: `mass_crawl.py` (rungs 1-3) · `camoufox_bridge.py` (rung 4) ·75`deep_crawl.py` (whole site).7677## Non-negotiables7879- `cache_mode=CacheMode.BYPASS` / `-bc` — default ENABLED serves stale pages and still reports success.80- One `AsyncWebCrawler` context for all URLs. Per-URL `async with` relaunches the browser (~50x slower).81- Check `result.success` per item — `arun_many` returns failures, never raises.82- Respect robots.txt + rate limits unless the target is the user's. Use `RateLimiter`, not a bare loop.
sergeyizmailov