Anti Bot Scraping

Use when a site blocks your scraper and you need to get past it — 403 Forbidden on the first request, JS challenges, CAPTCHAs, Cloudflare Turnstile, DataDome, Akamai, PerimeterX, "Just a moment...", access denied, IP bans, or an empty/skeleton page where data should be. Covers diagnosing transient failures vs real blocks, the cheapest-first technique ladder (plain HTTP → TLS fingerprint spoof → stealth browser → full browser), TLS spoofing with impit/curl_cffi, Camoufox stealth-browser setup, the Cloudflare Turnstile iframe-click, homepage warmup for cookie-based defenses, browser cookie reuse, XHR interception, proxy selection (residential vs datacenter vs SERP), and which targets genuinely resist affordable bypasses. Triggers on "site is blocking me", "getting 403", "Cloudflare", "DataDome", "Turnstile", "CAPTCHA", "bypass anti-bot", "scraper stopped working", "blocked".

thirdwatch-dev Updated

File contents

thirdwatch-dev/scraping-skills commit 9574cdea7b

Frequently asked questions

npx get-skill add thirdwatch-dev/anti-bot-scraping