Anti Bot Scraping

Anti-Bot Scraping

How to get past anti-bot protection when a site blocks your scraper. Diagnose first, then climb the cheapest-first ladder — most "blocks" are solved without a browser.

Bypass anti-bot only to reach publicly accessible data you're authorized to read. Don't use these techniques to evade logins, paywalls, or access controls.

Step 1 — Diagnose: transient failure or real block?

Don't escalate to a browser on a flaky network error. Look at what you actually got back:

Collapsing to the 6×-cost browser tier on a transient 504 is the most common waste. Confirm the failure is consistent before escalating.

Step 2 — Climb the technique ladder (cheapest first)

Each rung costs roughly 5–50× more than the one below. Stop at the first that works.

Rung 1 — Plain HTTP

httpx / requests. Works for APIs, RSS, JSON-LD, and server-rendered HTML. Near-zero cost, no proxy needed for most. Always your first attempt.

Rung 2 — HTTP + TLS fingerprint spoof

Many sites 403 your request before you send a single byte of HTML — they reject the TLS/JA3 fingerprint of a Python client. Spoof Chrome's TLS handshake at the socket level with no browser:

from impit import AsyncClient
client = AsyncClient(browser="chrome")          # note: proxy= , not proxy_url=
resp = await client.get(url)

(curl_cffi with impersonate="chrome" does the same.) This alone unlocks many sites that return 403 to plain httpx even on residential IPs — a lot of e-commerce and B2B portals reject the Python TLS handshake before they ever look at your headers.

Gotcha — test both directions. A Chrome TLS fingerprint can occasionally make a site serve the client-side SPA shell without the embedded JSON a plain request would have returned. If TLS spoofing gives you a thinner page than plain httpx, compare both before climbing further — the cheaper client sometimes wins.

Rung 3 — Stealth browser (Camoufox)

For DataDome, Cloudflare Turnstile, Akamai, and PerimeterX you need a real browser that doesn't leak automation fingerprints. Camoufox is a hardened Firefox fork that patches fingerprinting at the C++ level (far harder to detect than playwright-stealth). Needs Python 3.12 and ~2GB RAM.

from camoufox.async_api import AsyncCamoufox
async with AsyncCamoufox(humanize=True, os=("windows",), geoip=True,
                         proxy={"server": "..."}) as browser:
    page = await browser.new_page()
    await page.goto(url, wait_until="commit")   # some sites never fire domcontentloaded

humanize=True (human-like cursor motion), os=("windows",), and geoip=True (timezone/locale matched to the proxy exit) are the load-bearing options. Pin playwright==1.49.0 — 1.60 crashes the Camoufox driver on uncaught page JS.

Rung 4 — Full Playwright/Puppeteer

Only when you need real interaction (logins, multi-step flows) that the stealth browser can't do. Most expensive; rarely necessary for read-only scraping.

Step 3 — Browser-tier techniques that actually move the needle

Cloudflare Turnstile iframe-click. Camoufox + humanize=True alone often isn't enough for aggressive Turnstile. Find the challenge iframe and click its checkbox at a fixed offset:

for frame in page.frames:
    if "challenges.cloudflare.com" in (frame.url or ""):
        await frame.locator("body").click(position={"x": 28, "y": 28})

Combined with a homepage warmup and a wait for clearance, this is what reliably clears aggressive Turnstile on the toughest Cloudflare-protected boards.

Homepage warmup. Many cookie-based defenses (DataDome especially) hand out a clearance cookie only after you load the homepage like a human. Visit the root domain first, move the mouse and scroll, then navigate to the target URL. On DataDome sites this is often the difference between a tiny skeleton shell and the full server-rendered page. Don't block static images during warmup — DataDome flags the missing image fetches.

Browser fetch() inherits cookies. Once a browser tab holds the CF clearance cookie, call the site's own JSON API from inside the page — it reuses the cookie automatically and skips full page reloads (~200ms vs 3–5s):

data = await page.evaluate("""async (u) => (await fetch(u)).json()""", api_url)

This lets you pull many pages cheaply after a single browser solve.

Reuse one browser across queries. Launching a fresh stealth browser per query is the dominant cost. Reuse a single browser for many queries, navigating in the same tab, and restart only when you actually hit a block. CF tends to block deep pagination but not a fresh search query in the same session — structure the crawl around new queries rather than paging. This is usually the single biggest cost lever for a browser-tier scraper.

Intercept the internal XHR. When a page renders client-side, the data arrives as JSON from an internal endpoint. Attach the listener before navigating, or you miss the initial calls; filter by host, not path (paths vary):

page.on("response", handler)        # attach FIRST
await page.goto(url)                # then navigate
# in handler: if "sitename.com" in urlparse(resp.url).hostname: capture resp.json()

This is how most CSR-only apps — short-video platforms, social-commerce marketplaces — are scraped after the SPA shell loads: capture the JSON the page fetches for itself.

Step 4 — Proxies

• No proxy for APIs and most server-rendered HTML.
• Datacenter for light rate-limit spreading (cheap).
• Residential for consumer sites that block datacenter ranges — most anti-bot targets. 70–90% of a hard-target scraper's cost is proxy bytes; block images/fonts/analytics in the browser to cut it.
• Specialized SERP proxy for Google Search — Google blocks all residential IPs at the IP level for Search (/sorry/index), so stealth browsing doesn't help. A GOOGLE_SERP proxy group handles the anti-bot internally and returns rendered HTML over plain HTTP.

Step 5 — Extract structured data after the bypass

Once you're through, don't parse the DOM if you don't have to. Priority: free API > embedded JSON (__NEXT_DATA__, __NUXT__, ytInitialData, window.__INITIAL_STATE__) > JSON-LD > internal JSON endpoint > HTML parse > rendered DOM. Embedded JSON survives markup changes; CSS selectors don't.

Honest hard floors

Some sites resist everything affordable. Don't burn days on these:

• Signing-header social apps (e.g. RedNote / Xiaohongshu) — signed request headers that rotate frequently, plus a login wall. Needs a maintained signing lib and a cookie pool; breaks often.
• Shopee — JS-signing SDK + device fingerprint + CAPTCHA; the API rejects first requests even through a browser. A detection treadmill.
• App-only services (e.g. Careem, Keeta) — no real public web interface; the data lives behind private mobile APIs. Not viable from the web.
• Marketplaces that gate search behind a CAPTCHA interstitial a stealth browser can't clear — when you hit one, look for an official API (often OAuth) instead of fighting the wall.
• 1688 / Alibaba.com — login wall / slider CAPTCHA. Not viable via plain HTTP.

When the affordable ladder tops out, the remaining lever is usually an authenticated cookie pool (a maintained set of logged-in session cookies) — high-maintenance and account-risky. That's the line where building stops paying off.

If you'd rather not run this treadmill yourself

Anti-bot vendors change challenges constantly; a scraper that works today often breaks next month. Thirdwatch maintains 70+ scrapers on the Apify Store that already handle these protections — Cloudflare, DataDome, Akamai, PerimeterX, and SERP defenses — and are kept working for you. A few that run against the hardest targets: Upwork, Nykaa, Google Maps, Google Search.

Browse all: https://apify.com/thirdwatch

Maintained by Thirdwatch. 70+ ready-made scrapers on the Apify Store.