Data Acquisition & Web Scraping Toolkit
Tool Selection — Decision Tree
What are you downloading?
0. Quick probe — is this URL alive? What's on this page?
→ WebFetch (built-in, no setup) or Exa crawling
→ Use BEFORE building any pipeline
1. Direct file URL (CSV, ZIP, JSON, PDF)?
→ curl/requests first. If SSL fails → fallback chain below
→ If Cloudflare/bot detection → curl_cffi or Scrapfly
2. Page behind cookie-based login (Twitter, Instagram, Pinterest)?
→ chrome-cookies — extract session cookies from Chrome's DB, use with requests
→ Zero setup, no browser automation needed, works while Chrome is running
→ Falls back gracefully: if cookies expired, tells you to re-login
2b. Page behind SSO/Keycloak/Google login?
→ claude-in-chrome (user's live Chrome session) — ONLY reliable approach
→ STOP if you hit CAPTCHA, MFA, or login wall — ask the human
3. Page behind Cloudflare/anti-bot?
→ curl_cffi (TLS fingerprint impersonation) — try first, free
→ Scrapfly (asp=True) — paid fallback, handles JS rendering too
→ Browserbase — cloud browser, last resort for complex JS
4. Need structured data from a page (not just raw HTML)?
→ Firecrawl extract — pass JSON schema, get typed output
→ Firecrawl also good for site crawl/map (discover all URLs on a domain)
5. Need to render JavaScript?
→ Scrapfly with render_js=True
→ Browserbase (full cloud Chromium)
→ agent-browser local — only if site doesn't block automation
6. Need to interact (click, fill forms, navigate)?
→ claude-in-chrome for authenticated sites
→ Browserbase for non-authenticated complex flows
→ agent-browser local for simple non-protected sites (/agent-browser skill; snapshot → @eN refs)
7. Stuck after 2-3 attempts?
→ STOP. Tell the user what you tried and what failed.
→ Don't build increasingly elaborate workarounds.
→ The blocker might be access-tier (membership, license), not technical.
8. Software artifacts (git repos, model weights, packages)?
→ git clone for repos. huggingface-cli download for gated models.
→ Check LICENSE/access gates FIRST — tell user if signup needed.
→ For pip/uv packages: uv add, not pip install.
Operational Guardrails
Working directory: Always use absolute paths for download destinations. The Bash tool can reset cwd between calls. Use curl -o /absolute/path/file.csv not curl -o data/file.csv.
Large downloads (>100MB): Use run_in_background: true on the Bash tool call. Check progress with ls -lh /path/ while doing other work.
Bulk downloads: 0.5s between requests to same domain (government sites are polite but not infinite). sleep 0.5 between curl calls.
FRED shortcut: Many macro data series are mirrored on FRED. Direct CSV: curl -sL "https://fred.stlouisfed.org/graph/fredgraph.csv?id=SERIES_ID" -o series.csv
Tool Quick Reference
Per-tool details, code examples, and setup in references/.
| Tool |
When |
Free? |
Reference |
| chrome-cookies |
Cookie-based auth (Twitter, IG, Pinterest) — extract from Chrome DB |
Yes |
chrome-cookies.md |
| plain requests/curl |
Default first attempt, works ~80% |
Yes |
plain-requests.md |
| curl_cffi |
Cloudflare/TLS fingerprint blocks |
Yes |
curl-cffi.md |
| Scrapfly |
Anti-bot + JS rendering curl_cffi can't beat |
Paid |
scrapfly.md |
| Browserbase |
Complex multi-step JS flows (not SSO) |
Paid |
browserbase.md |
| claude-in-chrome |
ANY login/SSO — only reliable approach |
Free |
claude-in-chrome.md |
| WebFetch |
Quick URL probe (built-in) |
Free |
webfetch.md |
| Exa |
Search + content extraction via MCP |
Free |
exa.md |
| Firecrawl |
Structured extraction, site crawl/map |
Paid |
firecrawl.md |
| agent-browser |
Local headless browser, simple sites (replaced Playwright lane 2026-07-20) |
Free |
/agent-browser skill; agent-browser skills get core |
| Software artifacts |
Git repos, HF models, packages |
Free |
software-artifacts.md |
The Fallback Chain
Ordered by cost — free strategies first:
1. requests.get() — plain HTTP, works 80% of the time
2. requests.get(verify=False) — SSL certificate issues
3. curl --insecure — different TLS stack, catches edge cases
4. curl --tlsv1.2 --insecure — force TLS 1.2 for old servers
5. Scrapfly (asp=True) — paid, handles Cloudflare/anti-bot
Each strategy checks for HTML traps (got a landing page instead of data) and cleans up partial files on failure.
Per-Domain Gotchas
| Domain |
Issue |
Solution |
sec.gov |
Requires email in User-Agent, rate limit 10 req/sec |
User-Agent: project-name admin@email.com, 0.1s delay |
bls.gov |
Blocks default UA |
Use browser UA string |
data.cms.gov |
SPA redirects, HTML trap |
Check Content-Type, verify file isn't HTML |
census.gov |
Rate limits |
0.5s delay between requests |
lda.senate.gov |
Aggressive rate limit |
2.5s delay |
| ICPSR (Keycloak SSO) |
Cookies don't transfer, session is server-side |
claude-in-chrome only |
| Google accounts |
Blocks cloud browsers |
claude-in-chrome only |
What Does NOT Work (Don't Try)
| Approach |
Failure mode |
| browser_cookie3 for SSO sites |
Extracts cookies from Chrome's cookie DB, but server-side sessions don't transfer. Site sees empty session. For cookie-auth sites (Twitter, IG, Pinterest), use chrome-cookies module instead — same technique but maintained, tested, with auto-profile detection. |
Chrome --remote-debugging-port on macOS |
App Sandbox prevents port from opening. lsof shows nothing. |
| Playwright persistent context + Chrome profile |
Copies cookies/local storage but SSO session state lives server-side. Doesn't work for ICPSR, Google, etc. |
| Browserbase + Google SSO |
Google fingerprints cloud browser environments and blocks login. |
| Browserbase + Keycloak email login |
If account was created via Google SSO, there is no password. "Sign in with email" button exists but login fails. |
API Keys
All keys in .env.local at project root (gitignored). Details + load pattern: api-keys.md
Download Verification & QA
Always verify downloads — HTML traps, truncated files, wrong schemas are common. Full verification checklist, resumable download pattern, and format-specific profiling: download-verification.md
Anti-Patterns
- Don't build agent-browser/Playwright automation for SSO sites. Use claude-in-chrome.
- Don't retry a wall with fancier code. If the blocker is access-tier (not technical), stop coding.
- Don't accumulate probe/download scripts. Document the lesson, delete the script.
- Don't use
browser_cookie3. Use chrome_cookies module instead (selve/scripts/tools/chrome_cookies.py). Same Keychain+AES approach but maintained, with auto-profile detection. Neither works for SSO.
- Don't pay for Scrapfly/Browserbase before trying curl_cffi. Free first, paid last.
1---2name: data-acquisition-23description: Web scraping and data download toolkit — curl_cffi, Scrapfly, Firecrawl, Browserbase, claude-in-chrome, Exa, agent-browser. Covers which tool for which situation, API keys, fallback chains, structured extraction, authenticated session approaches, and what doesn't work on macOS. Use when downloading data, scraping websites, or automating browser interactions.4user-invocable: true5argument-hint: '[URL, site name, or scraping problem]'6effort: medium7metadata:8 source: https://skillmd.com/skills/markusstrasser/data-acquisition-29---1011# Data Acquisition & Web Scraping Toolkit1213## Tool Selection — Decision Tree1415```16What are you downloading?17180. Quick probe — is this URL alive? What's on this page?19 → WebFetch (built-in, no setup) or Exa crawling20 → Use BEFORE building any pipeline21221. Direct file URL (CSV, ZIP, JSON, PDF)?23 → curl/requests first. If SSL fails → fallback chain below24 → If Cloudflare/bot detection → curl_cffi or Scrapfly25262. Page behind cookie-based login (Twitter, Instagram, Pinterest)?27 → chrome-cookies — extract session cookies from Chrome's DB, use with requests28 → Zero setup, no browser automation needed, works while Chrome is running29 → Falls back gracefully: if cookies expired, tells you to re-login30312b. Page behind SSO/Keycloak/Google login?32 → claude-in-chrome (user's live Chrome session) — ONLY reliable approach33 → STOP if you hit CAPTCHA, MFA, or login wall — ask the human34353. Page behind Cloudflare/anti-bot?36 → curl_cffi (TLS fingerprint impersonation) — try first, free37 → Scrapfly (asp=True) — paid fallback, handles JS rendering too38 → Browserbase — cloud browser, last resort for complex JS39404. Need structured data from a page (not just raw HTML)?41 → Firecrawl extract — pass JSON schema, get typed output42 → Firecrawl also good for site crawl/map (discover all URLs on a domain)43445. Need to render JavaScript?45 → Scrapfly with render_js=True46 → Browserbase (full cloud Chromium)47 → agent-browser local — only if site doesn't block automation48496. Need to interact (click, fill forms, navigate)?50 → claude-in-chrome for authenticated sites51 → Browserbase for non-authenticated complex flows52 → agent-browser local for simple non-protected sites (/agent-browser skill; snapshot → @eN refs)53547. Stuck after 2-3 attempts?55 → STOP. Tell the user what you tried and what failed.56 → Don't build increasingly elaborate workarounds.57 → The blocker might be access-tier (membership, license), not technical.58598. Software artifacts (git repos, model weights, packages)?60 → git clone for repos. huggingface-cli download for gated models.61 → Check LICENSE/access gates FIRST — tell user if signup needed.62 → For pip/uv packages: uv add, not pip install.63```6465## Operational Guardrails6667**Working directory:** Always use absolute paths for download destinations. The Bash tool can reset cwd between calls. Use `curl -o /absolute/path/file.csv` not `curl -o data/file.csv`.6869**Large downloads (>100MB):** Use `run_in_background: true` on the Bash tool call. Check progress with `ls -lh /path/` while doing other work.7071**Bulk downloads:** 0.5s between requests to same domain (government sites are polite but not infinite). `sleep 0.5` between curl calls.7273**FRED shortcut:** Many macro data series are mirrored on FRED. Direct CSV: `curl -sL "https://fred.stlouisfed.org/graph/fredgraph.csv?id=SERIES_ID" -o series.csv`7475## Tool Quick Reference7677Per-tool details, code examples, and setup in `references/`.7879| Tool | When | Free? | Reference |80|------|------|-------|-----------|81| chrome-cookies | Cookie-based auth (Twitter, IG, Pinterest) — extract from Chrome DB | Yes | [chrome-cookies.md](references/chrome-cookies.md) |82| plain requests/curl | Default first attempt, works ~80% | Yes | [plain-requests.md](references/plain-requests.md) |83| curl_cffi | Cloudflare/TLS fingerprint blocks | Yes | [curl-cffi.md](references/curl-cffi.md) |84| Scrapfly | Anti-bot + JS rendering curl_cffi can't beat | Paid | [scrapfly.md](references/scrapfly.md) |85| Browserbase | Complex multi-step JS flows (not SSO) | Paid | [browserbase.md](references/browserbase.md) |86| claude-in-chrome | ANY login/SSO — only reliable approach | Free | [claude-in-chrome.md](references/claude-in-chrome.md) |87| WebFetch | Quick URL probe (built-in) | Free | [webfetch.md](references/webfetch.md) |88| Exa | Search + content extraction via MCP | Free | [exa.md](references/exa.md) |89| Firecrawl | Structured extraction, site crawl/map | Paid | [firecrawl.md](references/firecrawl.md) |90| agent-browser | Local headless browser, simple sites (replaced Playwright lane 2026-07-20) | Free | /agent-browser skill; `agent-browser skills get core` |91| Software artifacts | Git repos, HF models, packages | Free | [software-artifacts.md](references/software-artifacts.md) |9293## The Fallback Chain9495Ordered by cost — free strategies first:9697```981. requests.get() — plain HTTP, works 80% of the time992. requests.get(verify=False) — SSL certificate issues1003. curl --insecure — different TLS stack, catches edge cases1014. curl --tlsv1.2 --insecure — force TLS 1.2 for old servers1025. Scrapfly (asp=True) — paid, handles Cloudflare/anti-bot103```104105Each strategy checks for **HTML traps** (got a landing page instead of data) and cleans up partial files on failure.106107## Per-Domain Gotchas108109| Domain | Issue | Solution |110|--------|-------|----------|111| `sec.gov` | Requires email in User-Agent, rate limit 10 req/sec | `User-Agent: project-name admin@email.com`, 0.1s delay |112| `bls.gov` | Blocks default UA | Use browser UA string |113| `data.cms.gov` | SPA redirects, HTML trap | Check Content-Type, verify file isn't HTML |114| `census.gov` | Rate limits | 0.5s delay between requests |115| `lda.senate.gov` | Aggressive rate limit | 2.5s delay |116| ICPSR (Keycloak SSO) | Cookies don't transfer, session is server-side | claude-in-chrome only |117| Google accounts | Blocks cloud browsers | claude-in-chrome only |118119## What Does NOT Work (Don't Try)120121| Approach | Failure mode |122|----------|-------------|123| **browser_cookie3 for SSO sites** | Extracts cookies from Chrome's cookie DB, but server-side sessions don't transfer. Site sees empty session. For cookie-auth sites (Twitter, IG, Pinterest), use chrome-cookies module instead — same technique but maintained, tested, with auto-profile detection. |124| **Chrome `--remote-debugging-port` on macOS** | App Sandbox prevents port from opening. `lsof` shows nothing. |125| **Playwright persistent context + Chrome profile** | Copies cookies/local storage but SSO session state lives server-side. Doesn't work for ICPSR, Google, etc. |126| **Browserbase + Google SSO** | Google fingerprints cloud browser environments and blocks login. |127| **Browserbase + Keycloak email login** | If account was created via Google SSO, there is no password. "Sign in with email" button exists but login fails. |128129## API Keys130131All keys in `.env.local` at project root (gitignored). Details + load pattern: [api-keys.md](references/api-keys.md)132133## Download Verification & QA134135Always verify downloads — HTML traps, truncated files, wrong schemas are common. Full verification checklist, resumable download pattern, and format-specific profiling: [download-verification.md](references/download-verification.md)136137## Anti-Patterns1381391. **Don't build agent-browser/Playwright automation for SSO sites.** Use claude-in-chrome.1402. **Don't retry a wall with fancier code.** If the blocker is access-tier (not technical), stop coding.1413. **Don't accumulate probe/download scripts.** Document the lesson, delete the script.1424. **Don't use `browser_cookie3`.** Use `chrome_cookies` module instead (selve/scripts/tools/chrome_cookies.py). Same Keychain+AES approach but maintained, with auto-profile detection. Neither works for SSO.1435. **Don't pay for Scrapfly/Browserbase before trying curl_cffi.** Free first, paid last.