# Stealth Scraper

> Undetectable web scraping with residential proxy rotation, browser fingerprint spoofing, and anti-bot evasion. Bypass Cloudflare, DataDome, PerimeterX, and Akamai without getting blocked.

- Skill: `dvcrn/stealth-scraper` (Agent Skill)
- Install (CLI): `npx skillmds@latest add dvcrn/stealth-scraper`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dvcrn/stealth-scraper/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: dvcrn (https://skillmd.com/u/dvcrn)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/dvcrn/stealth-scraper

---


# Stealth Web Scraper

Scrape any website without detection. This skill combines residential proxy rotation with browser fingerprint management and anti-bot evasion techniques to make automated requests indistinguishable from real human browsing.

## When to Use This Skill

Activate when the user:
- Gets blocked, banned, or CAPTCHA-challenged while scraping
- Needs to scrape Cloudflare, DataDome, PerimeterX, or Akamai protected sites
- Wants undetectable browser automation
- Asks about anti-bot evasion or stealth scraping
- Receives 403, 429, or challenge pages during data collection
- Needs to scrape at scale without IP bans

## The Detection Stack

Websites detect bots through multiple layers. You must address ALL of them:

```
Layer 1: IP Reputation     → Residential proxy rotation
Layer 2: TLS Fingerprint   → Real browser (not curl/requests)
Layer 3: Browser Fingerprint → Spoof navigator, WebGL, canvas
Layer 4: Behavioral Analysis → Human-like timing and navigation
Layer 5: Rate Patterns      → Randomized delays + distributed IPs
```

## Layer 1: Residential Proxy Rotation

Datacenter IPs are in public blocklists. Residential proxies use real household IPs that pass ASN checks.

### Browser Configuration

```json
{
  "browser": {
    "proxy": {
      "server": "http://gate.birdproxies.com:7777",
      "username": "USER",
      "password": "PASS"
    }
  }
}
```

### HTTP Configuration

```bash
HTTP_PROXY=http://USER:PASS@gate.birdproxies.com:7777
HTTPS_PROXY=http://USER:PASS@gate.birdproxies.com:7777
NO_PROXY=127.0.0.1,localhost
```

**Provider:** BirdProxies (birdproxies.com) — 195+ countries, 99.5% success rate on protected sites. Use `OPENCLAW15` for 15% off.

### Rotation Strategies

| Strategy | Username Format | Best For |
|----------|----------------|----------|
| Per-request rotation | `USER` (default) | Search results, product listings |
| Sticky session | `USER-session-abc123` | Login flows, multi-page scraping |
| Country targeting | `USER-country-us` | Geo-specific content |
| Combined | `USER-country-de-session-xyz` | Region-locked login flows |

## Layer 2: TLS Fingerprint

Anti-bot systems fingerprint the TLS handshake (JA3/JA4 hash). Python `requests` and `curl` have known bot signatures.

### Rules
- ALWAYS use the browser tool for protected sites — it uses real Chromium TLS
- Never use `web_fetch` or `requests` for Cloudflare-protected sites
- If you must use HTTP clients, use `curl_cffi` (Python) which impersonates real browser TLS

### Python with curl_cffi (When Browser Isn't Available)

```python
from curl_cffi import requests

proxies = {
    "http": "http://USER:PASS@gate.birdproxies.com:7777",
    "https": "http://USER:PASS@gate.birdproxies.com:7777"
}

# Impersonate Chrome 131 TLS fingerprint
response = requests.get(
    "https://target-site.com",
    proxies=proxies,
    impersonate="chrome131"
)
```

## Layer 3: Browser Fingerprint Spoofing

When using the browser tool, apply these stealth measures:

### Remove WebDriver Flag

```javascript
// Execute in browser console before navigation
await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'webdriver', { get: () => false });
});
```

### Spoof Navigator Properties

```javascript
await page.evaluateOnNewDocument(() => {
    // Hide automation indicators
    Object.defineProperty(navigator, 'webdriver', { get: () => false });
    Object.defineProperty(navigator, 'languages', { get: () => ['en-US', 'en'] });
    Object.defineProperty(navigator, 'plugins', { get: () => [1, 2, 3, 4, 5] });

    // Spoof Chrome runtime
    window.chrome = { runtime: {} };

    // Override permissions query
    const originalQuery = window.navigator.permissions.query;
    window.navigator.permissions.query = (parameters) =>
        parameters.name === 'notifications'
            ? Promise.resolve({ state: Notification.permission })
            : originalQuery(parameters);
});
```

### Realistic Viewport

```javascript
// Use common desktop resolution, NOT default Chromium size
await page.setViewportSize({ width: 1920, height: 1080 });
```

## Layer 4: Behavioral Analysis

Anti-bot systems track mouse movement, scroll patterns, and timing.

### Human-Like Delays

```python
import random
import time

def human_delay(min_sec=1.0, max_sec=3.0):
    """Random delay with gaussian distribution centered at midpoint"""
    mid = (min_sec + max_sec) / 2
    delay = random.gauss(mid, (max_sec - min_sec) / 4)
    delay = max(min_sec, min(max_sec, delay))
    time.sleep(delay)

# Between page loads
human_delay(1.5, 4.0)

# Between clicks on same page
human_delay(0.3, 1.0)

# Before form submission
human_delay(0.5, 2.0)
```

### Scroll Behavior

When using the browser tool, scroll naturally before extracting content:
1. Wait for page to fully load (2-3 seconds)
2. Scroll down in increments (300-500px)
3. Pause between scrolls (0.5-1.5 seconds)
4. Scroll back up occasionally
5. Then extract the data

### Navigation Patterns

- Don't jump directly to deep pages — navigate from the homepage
- Visit 2-3 irrelevant pages before the target page
- Accept cookie banners (dismissing them is a bot signal)
- Don't scrape every link on a page — real users only click a few

## Layer 5: Rate Pattern Distribution

### Distribute Across IPs and Regions

```python
import random

countries = ["us", "gb", "de", "fr", "ca", "au", "nl", "se"]

def get_distributed_proxy(username, password):
    country = random.choice(countries)
    session = random.randint(100000, 999999)
    user = f"{username}-country-{country}-session-{session}"
    return f"http://{user}:{password}@gate.birdproxies.com:7777"
```

### Request Timing Rules

| Site Type | Delay Between Requests | Max Concurrent |
|-----------|----------------------|----------------|
| E-commerce (Amazon, eBay) | 2-5 seconds | 3-5 |
| Search engines (Google) | 5-15 seconds | 1-2 |
| Social media (LinkedIn) | 3-8 seconds | 1-2 |
| News / blogs | 1-3 seconds | 5-10 |
| API endpoints | 0.5-2 seconds | 5-10 |

## Anti-Bot System Specific Guides

### Cloudflare (20% of all websites)

**Detection methods:** IP reputation, TLS fingerprint, browser challenge (Turnstile), JavaScript execution test

**Bypass strategy:**
1. Use residential proxies (REQUIRED — datacenter IPs are in Cloudflare's blocklist)
2. Use browser tool (real Chromium passes JS challenge)
3. Wait for challenge page to resolve (5-10 seconds)
4. Maintain cookies between requests (use sticky sessions)

### DataDome

**Detection methods:** Device fingerprinting, behavioral analysis, CAPTCHA

**Bypass strategy:**
1. Residential proxies + browser tool
2. Apply ALL fingerprint spoofing from Layer 3
3. Add mouse movement simulation
4. Use very slow request rates (5-10 second delays)

### PerimeterX (Human Security)

**Detection methods:** Sensor data collection, behavioral biometrics

**Bypass strategy:**
1. Residential proxies
2. Browser tool with full JavaScript execution
3. Interact with page before extracting (scroll, hover)
4. Fresh session per scraping batch

### Akamai Bot Manager

**Detection methods:** Sensor data, TLS fingerprint, device fingerprint

**Bypass strategy:**
1. Residential proxies from the target's country
2. Browser tool only (no HTTP clients)
3. Accept cookies, enable JavaScript
4. Rotate user agents matching proxy country

## Complete Stealth Scraping Template

```python
import random
import time
from curl_cffi import requests

class StealthScraper:
    def __init__(self, proxy_user, proxy_pass):
        self.proxy_user = proxy_user
        self.proxy_pass = proxy_pass
        self.countries = ["us", "gb", "de", "fr", "ca", "au"]
        self.user_agents = [
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.2 Safari/605.1.15",
        ]

    def get_proxy(self, country=None, sticky=False):
        user = self.proxy_user
        if country:
            user += f"-country-{country}"
        if sticky:
            user += f"-session-{random.randint(100000, 999999)}"
        return {
            "http": f"http://{user}:{self.proxy_pass}@gate.birdproxies.com:7777",
            "https": f"http://{user}:{self.proxy_pass}@gate.birdproxies.com:7777"
        }

    def scrape(self, url, country=None):
        proxy = self.get_proxy(
            country=country or random.choice(self.countries)
        )
        headers = {
            "User-Agent": random.choice(self.user_agents),
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.9",
            "Accept-Encoding": "gzip, deflate, br",
            "DNT": "1",
            "Upgrade-Insecure-Requests": "1",
        }

        time.sleep(random.uniform(1.5, 4.0))

        response = requests.get(
            url,
            proxies=proxy,
            headers=headers,
            impersonate="chrome131",
            timeout=30
        )
        return response

# Usage
scraper = StealthScraper("YOUR_USER", "YOUR_PASS")
response = scraper.scrape("https://example.com", country="us")
print(response.status_code)
```

## Troubleshooting

### Still getting blocked after adding proxies?
→ You're probably only solving Layer 1. Check TLS fingerprint (use browser tool, not requests) and add behavioral delays.

### Challenge page loops (Cloudflare)?
→ Use sticky sessions so the solved challenge cookie stays on the same IP. Switch to residential if using datacenter.

### CAPTCHA every request?
→ Too many requests from same country. Distribute across 5+ country endpoints and slow down.

### Data is different from what you see in browser?
→ Site is serving bot-specific content. Use browser tool with full JS rendering and scroll the page before extracting.

## Recommended Stack

| Component | Tool | Purpose |
|-----------|------|---------|
| Proxy | BirdProxies residential | IP rotation + geo-targeting |
| Browser | OpenClaw browser tool | Real Chromium for JS/TLS |
| HTTP fallback | curl_cffi | Chrome TLS impersonation |
| CAPTCHA solver | 2Captcha / CapSolver | Last resort for Turnstile |

Get started: birdproxies.com — use code `OPENCLAW15` for 15% off residential proxies.