Data Acquisition & Web Scraping Toolkit
Tool Selection — Decision Tree
What are you downloading?
0. Quick probe — is this URL alive? What's on this page?
→ WebFetch (built-in, no setup) or Exa crawling
→ Use BEFORE building any pipeline
1. Direct file URL (CSV, ZIP, JSON, PDF)?
→ curl/requests first. If SSL fails → fallback chain below
→ If Cloudflare/bot detection → curl_cffi or Scrapfly
2. Page behind cookie-based login (Twitter, Instagram, Pinterest)?
→ chrome-cookies — extract session cookies from Chrome's DB, use with requests
→ Zero setup, no browser automation needed, works while Chrome is running
→ Falls back gracefully: if cookies expired, tells you to re-login
2b. Page behind SSO/Keycloak/Google login?
→ claude-in-chrome (user's live Chrome session) — ONLY reliable approach
→ STOP if you hit CAPTCHA, MFA, or login wall — ask the human
3. Page behind Cloudflare/anti-bot?
→ curl_cffi (TLS fingerprint impersonation) — try first, free
→ Scrapfly (asp=True) — paid fallback, handles JS rendering too
→ Browserbase — cloud browser, last resort for complex JS
4. Need structured data from a page (not just raw HTML)?
→ Firecrawl extract — pass JSON schema, get typed output
→ Firecrawl also good for site crawl/map (discover all URLs on a domain)
5. Need to render JavaScript?
→ Scrapfly with render_js=True
→ Browserbase (full cloud Chromium)
→ agent-browser local — only if site doesn't block automation
6. Need to interact (click, fill forms, navigate)?
→ claude-in-chrome for authenticated sites
→ Browserbase for non-authenticated complex flows
→ agent-browser local for simple non-protected sites (/agent-browser skill; snapshot → @eN refs)
7. Stuck after 2-3 attempts?
→ STOP. Tell the user what you tried and what failed.
→ Don't build increasingly elaborate workarounds.
→ The blocker might be access-tier (membership, license), not technical.
8. Software artifacts (git repos, model weights, packages)?
→ git clone for repos. huggingface-cli download for gated models.
→ Check LICENSE/access gates FIRST — tell user if signup needed.
→ For pip/uv packages: uv add, not pip install.
Operational Guardrails
Working directory: Always use absolute paths for download destinations. The Bash tool can reset cwd between calls. Use curl -o /absolute/path/file.csv not curl -o data/file.csv.
Large downloads (>100MB): Use run_in_background: true on the Bash tool call. Check progress with ls -lh /path/ while doing other work.
Bulk downloads: 0.5s between requests to same domain (government sites are polite but not infinite). sleep 0.5 between curl calls.
FRED shortcut: Many macro data series are mirrored on FRED. Direct CSV: curl -sL "https://fred.stlouisfed.org/graph/fredgraph.csv?id=SERIES_ID" -o series.csv
Tool Quick Reference
Per-tool details, code examples, and setup in references/.
| Tool |
When |
Free? |
Reference |
| chrome-cookies |
Cookie-based auth (Twitter, IG, Pinterest) — extract from Chrome DB |
Yes |
chrome-cookies.md |
| plain requests/curl |
Default first attempt, works ~80% |
Yes |
plain-requests.md |
| curl_cffi |
Cloudflare/TLS fingerprint blocks |
Yes |
curl-cffi.md |
| Scrapfly |
Anti-bot + JS rendering curl_cffi can't beat |
Paid |
scrapfly.md |
| Browserbase |
Complex multi-step JS flows (not SSO) |
Paid |
browserbase.md |
| claude-in-chrome |
ANY login/SSO — only reliable approach |
Free |
claude-in-chrome.md |
| WebFetch |
Quick URL probe (built-in) |
Free |
webfetch.md |
| Exa |
Search + content extraction via MCP |
Free |
exa.md |
| Firecrawl |
Structured extraction, site crawl/map |
Paid |
firecrawl.md |
| agent-browser |
Local headless browser, simple sites (replaced Playwright lane 2026-07-20) |
Free |
/agent-browser skill; agent-browser skills get core |
| Software artifacts |
Git repos, HF models, packages |
Free |
software-artifacts.md |
The Fallback Chain
Ordered by cost — free strategies first:
1. requests.get() — plain HTTP, works 80% of the time
2. requests.get(verify=False) — SSL certificate issues
3. curl --insecure — different TLS stack, catches edge cases
4. curl --tlsv1.2 --insecure — force TLS 1.2 for old servers
5. Scrapfly (asp=True) — paid, handles Cloudflare/anti-bot
Each strategy checks for HTML traps (got a landing page instead of data) and cleans up partial files on failure.
Per-Domain Gotchas
| Domain |
Issue |
Solution |
sec.gov |
Requires email in User-Agent, rate limit 10 req/sec |
User-Agent: project-name admin@email.com, 0.1s delay |
bls.gov |
Blocks default UA |
Use browser UA string |
data.cms.gov |
SPA redirects, HTML trap |
Check Content-Type, verify file isn't HTML |
census.gov |
Rate limits |
0.5s delay between requests |
lda.senate.gov |
Aggressive rate limit |
2.5s delay |
| ICPSR (Keycloak SSO) |
Cookies don't transfer, session is server-side |
claude-in-chrome only |
| Google accounts |
Blocks cloud browsers |
claude-in-chrome only |
What Does NOT Work (Don't Try)
| Approach |
Failure mode |
| browser_cookie3 for SSO sites |
Extracts cookies from Chrome's cookie DB, but server-side sessions don't transfer. Site sees empty session. For cookie-auth sites (Twitter, IG, Pinterest), use chrome-cookies module instead — same technique but maintained, tested, with auto-profile detection. |
Chrome --remote-debugging-port on macOS |
App Sandbox prevents port from opening. lsof shows nothing. |
| Playwright persistent context + Chrome profile |
Copies cookies/local storage but SSO session state lives server-side. Doesn't work for ICPSR, Google, etc. |
| Browserbase + Google SSO |
Google fingerprints cloud browser environments and blocks login. |
| Browserbase + Keycloak email login |
If account was created via Google SSO, there is no password. "Sign in with email" button exists but login fails. |
API Keys
All keys in .env.local at project root (gitignored). Details + load pattern: api-keys.md
Download Verification & QA
Always verify downloads — HTML traps, truncated files, wrong schemas are common. Full verification checklist, resumable download pattern, and format-specific profiling: download-verification.md
Anti-Patterns
- Don't build agent-browser/Playwright automation for SSO sites. Use claude-in-chrome.
- Don't retry a wall with fancier code. If the blocker is access-tier (not technical), stop coding.
- Don't accumulate probe/download scripts. Document the lesson, delete the script.
- Don't use
browser_cookie3. Use chrome_cookies module instead (selve/scripts/tools/chrome_cookies.py). Same Keychain+AES approach but maintained, with auto-profile detection. Neither works for SSO.
- Don't pay for Scrapfly/Browserbase before trying curl_cffi. Free first, paid last.
1---2name: data-acquisition-23description: Web scraping and data download toolkit — curl_cffi, Scrapfly, Firecrawl, Browserbase, claude-in-chrome, Exa, agent-browser. Covers which tool for which situation, API keys, fallback chains, structured extraction, authenticated session approaches, and what doesn't work on macOS. Use when downloading data, scraping websites, or automating browser interactions.4---56# Data Acquisition & Web Scraping Toolkit78## Tool Selection — Decision Tree910```11What are you downloading?12130. Quick probe — is this URL alive? What's on this page?14 → WebFetch (built-in, no setup) or Exa crawling15 → Use BEFORE building any pipeline16171. Direct file URL (CSV, ZIP, JSON, PDF)?18 → curl/requests first. If SSL fails → fallback chain below19 → If Cloudflare/bot detection → curl_cffi or Scrapfly20212. Page behind cookie-based login (Twitter, Instagram, Pinterest)?22 → chrome-cookies — extract session cookies from Chrome's DB, use with requests23 → Zero setup, no browser automation needed, works while Chrome is running24 → Falls back gracefully: if cookies expired, tells you to re-login25262b. Page behind SSO/Keycloak/Google login?27 → claude-in-chrome (user's live Chrome session) — ONLY reliable approach28 → STOP if you hit CAPTCHA, MFA, or login wall — ask the human29303. Page behind Cloudflare/anti-bot?31 → curl_cffi (TLS fingerprint impersonation) — try first, free32 → Scrapfly (asp=True) — paid fallback, handles JS rendering too33 → Browserbase — cloud browser, last resort for complex JS34354. Need structured data from a page (not just raw HTML)?36 → Firecrawl extract — pass JSON schema, get typed output37 → Firecrawl also good for site crawl/map (discover all URLs on a domain)38395. Need to render JavaScript?40 → Scrapfly with render_js=True41 → Browserbase (full cloud Chromium)42 → agent-browser local — only if site doesn't block automation43446. Need to interact (click, fill forms, navigate)?45 → claude-in-chrome for authenticated sites46 → Browserbase for non-authenticated complex flows47 → agent-browser local for simple non-protected sites (/agent-browser skill; snapshot → @eN refs)48497. Stuck after 2-3 attempts?50 → STOP. Tell the user what you tried and what failed.51 → Don't build increasingly elaborate workarounds.52 → The blocker might be access-tier (membership, license), not technical.53548. Software artifacts (git repos, model weights, packages)?55 → git clone for repos. huggingface-cli download for gated models.56 → Check LICENSE/access gates FIRST — tell user if signup needed.57 → For pip/uv packages: uv add, not pip install.58```5960## Operational Guardrails6162**Working directory:** Always use absolute paths for download destinations. The Bash tool can reset cwd between calls. Use `curl -o /absolute/path/file.csv` not `curl -o data/file.csv`.6364**Large downloads (>100MB):** Use `run_in_background: true` on the Bash tool call. Check progress with `ls -lh /path/` while doing other work.6566**Bulk downloads:** 0.5s between requests to same domain (government sites are polite but not infinite). `sleep 0.5` between curl calls.6768**FRED shortcut:** Many macro data series are mirrored on FRED. Direct CSV: `curl -sL "https://fred.stlouisfed.org/graph/fredgraph.csv?id=SERIES_ID" -o series.csv`6970## Tool Quick Reference7172Per-tool details, code examples, and setup in `references/`.7374| Tool | When | Free? | Reference |75|------|------|-------|-----------|76| chrome-cookies | Cookie-based auth (Twitter, IG, Pinterest) — extract from Chrome DB | Yes | [chrome-cookies.md](references/chrome-cookies.md) |77| plain requests/curl | Default first attempt, works ~80% | Yes | [plain-requests.md](references/plain-requests.md) |78| curl_cffi | Cloudflare/TLS fingerprint blocks | Yes | [curl-cffi.md](references/curl-cffi.md) |79| Scrapfly | Anti-bot + JS rendering curl_cffi can't beat | Paid | [scrapfly.md](references/scrapfly.md) |80| Browserbase | Complex multi-step JS flows (not SSO) | Paid | [browserbase.md](references/browserbase.md) |81| claude-in-chrome | ANY login/SSO — only reliable approach | Free | [claude-in-chrome.md](references/claude-in-chrome.md) |82| WebFetch | Quick URL probe (built-in) | Free | [webfetch.md](references/webfetch.md) |83| Exa | Search + content extraction via MCP | Free | [exa.md](references/exa.md) |84| Firecrawl | Structured extraction, site crawl/map | Paid | [firecrawl.md](references/firecrawl.md) |85| agent-browser | Local headless browser, simple sites (replaced Playwright lane 2026-07-20) | Free | /agent-browser skill; `agent-browser skills get core` |86| Software artifacts | Git repos, HF models, packages | Free | [software-artifacts.md](references/software-artifacts.md) |8788## The Fallback Chain8990Ordered by cost — free strategies first:9192```931. requests.get() — plain HTTP, works 80% of the time942. requests.get(verify=False) — SSL certificate issues953. curl --insecure — different TLS stack, catches edge cases964. curl --tlsv1.2 --insecure — force TLS 1.2 for old servers975. Scrapfly (asp=True) — paid, handles Cloudflare/anti-bot98```99100Each strategy checks for **HTML traps** (got a landing page instead of data) and cleans up partial files on failure.101102## Per-Domain Gotchas103104| Domain | Issue | Solution |105|--------|-------|----------|106| `sec.gov` | Requires email in User-Agent, rate limit 10 req/sec | `User-Agent: project-name admin@email.com`, 0.1s delay |107| `bls.gov` | Blocks default UA | Use browser UA string |108| `data.cms.gov` | SPA redirects, HTML trap | Check Content-Type, verify file isn't HTML |109| `census.gov` | Rate limits | 0.5s delay between requests |110| `lda.senate.gov` | Aggressive rate limit | 2.5s delay |111| ICPSR (Keycloak SSO) | Cookies don't transfer, session is server-side | claude-in-chrome only |112| Google accounts | Blocks cloud browsers | claude-in-chrome only |113114## What Does NOT Work (Don't Try)115116| Approach | Failure mode |117|----------|-------------|118| **browser_cookie3 for SSO sites** | Extracts cookies from Chrome's cookie DB, but server-side sessions don't transfer. Site sees empty session. For cookie-auth sites (Twitter, IG, Pinterest), use chrome-cookies module instead — same technique but maintained, tested, with auto-profile detection. |119| **Chrome `--remote-debugging-port` on macOS** | App Sandbox prevents port from opening. `lsof` shows nothing. |120| **Playwright persistent context + Chrome profile** | Copies cookies/local storage but SSO session state lives server-side. Doesn't work for ICPSR, Google, etc. |121| **Browserbase + Google SSO** | Google fingerprints cloud browser environments and blocks login. |122| **Browserbase + Keycloak email login** | If account was created via Google SSO, there is no password. "Sign in with email" button exists but login fails. |123124## API Keys125126All keys in `.env.local` at project root (gitignored). Details + load pattern: [api-keys.md](references/api-keys.md)127128## Download Verification & QA129130Always verify downloads — HTML traps, truncated files, wrong schemas are common. Full verification checklist, resumable download pattern, and format-specific profiling: [download-verification.md](references/download-verification.md)131132## Anti-Patterns1331341. **Don't build agent-browser/Playwright automation for SSO sites.** Use claude-in-chrome.1352. **Don't retry a wall with fancier code.** If the blocker is access-tier (not technical), stop coding.1363. **Don't accumulate probe/download scripts.** Document the lesson, delete the script.1374. **Don't use `browser_cookie3`.** Use `chrome_cookies` module instead (selve/scripts/tools/chrome_cookies.py). Same Keychain+AES approach but maintained, with auto-profile detection. Neither works for SSO.1385. **Don't pay for Scrapfly/Browserbase before trying curl_cffi.** Free first, paid last.
markusstrasser