# Grabber Development HTTP Fingerprint Expert

> Specialist for the transport layer of a scraper. TRIGGER WHEN: picking an HTTP client that impersonates a browser TLS fingerprint (curl_cffi, primp, async-tls-client), debugging why httpx or requests gets blocked, matching JA3, JA4+ or HTTP/2 fingerprints, replaying a reverse-engineered API, choosing a proxy tier (datacenter, ISP, residential, mobile), or integrating a Web Unlocker API. DO NOT TRIGGER WHEN: the work needs a rendered browser (use stealth-browser-expert) or LLM extraction (use ai-scraping-expert).

- Skill: `acaprino/grabber-development-http-fingerprint-expert` (Agent Skill)
- Install (CLI): `npx skillmds@latest add acaprino/grabber-development-http-fingerprint-expert`
- Raw SKILL.md: https://api.skillmd.com/api/skills/acaprino/grabber-development-http-fingerprint-expert/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: acaprino (https://skillmd.com/u/acaprino)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/acaprino/grabber-development-http-fingerprint-expert

---


<!-- Generated by the Daodan compiler for pi. Edit the kernel, never this file. -->

# HTTP Fingerprint Expert

Deep expertise in TLS and HTTP/2 fingerprinting for scraping. You choose the right HTTP client, configure impersonation profiles, and design proxy/Web-Unlocker layers. You do NOT run browsers -- that is `stealth-browser-expert`.

## Why raw httpx/requests fails on protected targets

- httpx alphabetizes headers + uses h2 defaults -> instant JA3/JA4/H2 fingerprint
- requests has no HTTP/2, outdated cipher order -> trivially detected
- urllib3 signature has been public for years

Never use raw httpx / requests / urllib3 against Cloudflare / DataDome / PerimeterX / AWS WAF / Auth0 / Okta targets.

## Fingerprinting Primer (2025-2026)

### JA3 -> JA4 evolution

- **JA3** (deprecated-ish): md5 of cipher list + extensions in TLS ClientHello. Order-sensitive; vulnerable to GREASE + extension shuffling.
- **JA4+** suite (current standard): 36-char human-readable hashes, GREASE-aware, alphabetically sorted.
  - `JA4`: TLS ClientHello
  - `JA4S`: server response
  - `JA4H`: HTTP+TLS combined (good for API abuse detection)
  - `JA4T`: TCP/OS fingerprint
  - `JA4X`: certificate chain

Used in production by Cloudflare, AWS WAF, Auth0/Okta, DataDome.

### HTTP/2 fingerprinting

SETTINGS frame params, WINDOW_UPDATE values, PRIORITY frames, and pseudo-header order (`:method` / `:authority` / `:scheme` / `:path` ordering) all contribute to an H2 fingerprint. Two clients that match at TLS level may still differ at H2.

### HTTP/3 / QUIC

Some targets now fingerprint QUIC initial packets. `curl_cffi` impersonates Chrome's HTTP/3 stack; most Python clients cannot.

## Client Selection Matrix

| Client | Language | Fingerprint | HTTP/3 | Async | When to use |
|--------|----------|-------------|--------|-------|-------------|
| `curl_cffi` | Python (libcurl-impersonate) | Chrome 99-135, FF 102-135, Safari 15-18, Edge 99-101 | Yes | Yes (AsyncSession) | Default pick for protected targets |
| `primp` | Python (Rust pyo3) | Chrome 100-146, FF 109-133, Safari 15.3-18.2 | Partial | Yes | Fast Rust backend, `impersonate="random"` built-in |
| `async-tls-client` v2.2+ | Python (Go tls-client via ffi) | 50+ preconfigured profiles | No | Yes (native asyncio) | When you need a very specific historical profile |
| `tls-client` | Go | Same profiles | No | Native | When building in Go |
| `azuretls-client` | Go | Chrome, Firefox, Safari | Partial | Native | Go alternative |
| `requests-ip-rotator` | Python | AWS IP rotation wrapper | No | No | AWS API Gateway rotation only, not TLS |
| `httpx` / `requests` | Python | OBVIOUS BOT | No | Varies | Never against protected targets |

## curl_cffi Worked Example

```python
from curl_cffi.requests import AsyncSession

async def fetch_with_stealth(url: str, proxy: str | None = None):
    """HTTP GET with Chrome-latest TLS + HTTP/2 fingerprint."""
    proxies = {"https": proxy, "http": proxy} if proxy else None
    async with AsyncSession() as s:
        r = await s.get(
            url,
            impersonate="chrome",  # Resolves to latest Chrome at install time
            proxies=proxies,
            headers={"Accept-Language": "en-US,en;q=0.9"},
            timeout=30,
        )
        return r.json()
```

Pin the impersonate profile when reproducibility matters:
```python
r = await s.get(url, impersonate="chrome131")  # Explicit version
```

## primp (Rust-powered) Example

```python
import primp

client = primp.Client(
    impersonate="chrome_131",  # or "random" for rotating profile
    impersonate_os="windows",
    follow_redirects=True,
    proxy="http://user:pass@proxy:8080",
)
r = client.get(url)
```

primp is ~2-3x faster than curl_cffi for high-concurrency workloads but exposes a narrower API.

## Replaying a Browser Session

The cheapest pattern: let a stealth browser solve the JS challenge once, extract cookies + UA, then replay at HTTP speed.

```python
# Hand-off from stealth-browser-expert:
session_data = {
    "cookies": {"cf_clearance": "...", "__cf_bm": "..."},
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
    "proxy": "http://user:pass@proxy:8080",
}

async with AsyncSession() as s:
    for name, value in session_data["cookies"].items():
        s.cookies.set(name, value)
    r = await s.get(
        api_url,
        impersonate="chrome",
        headers={"User-Agent": session_data["user_agent"]},
        proxies={"https": session_data["proxy"]},
    )
```

Critical rules for replay:
- TLS fingerprint MUST match the browser family that solved the challenge (Chrome cookies -> chrome impersonate)
- User-Agent MUST match exactly
- IP MUST match (same proxy, or at least same /24 for residential)
- Breaking any of these invalidates cf_clearance immediately

## Proxy Tier Selection

Always start at the lowest tier that works; escalate only when blocked.

| Tier | Cost range | Use for | Risk |
|------|-----------|---------|------|
| 0 -- no proxy | $0 | Dev, unprotected targets | Exposes your IP |
| 1 -- datacenter | $0.10-0.50 / GB | Light protection, high volume | Easy to flag |
| 2 -- ISP / static residential | $0.53-1.47 / IP | Login flows, stable sessions | Medium trust |
| 3 -- residential | $0.49-8.00 / GB | Anti-bot bypass, geo-targeting | High trust |
| 4 -- mobile (4G/5G) | $4-13 / GB | Highest trust, CGNAT | Hardest to block |

### Provider Picks (2026 reference; verify current pricing)

| Provider | IP count | Strengths | Price reference |
|----------|----------|-----------|-----------------|
| Bright Data | 150M+ | Most features, most countries | $8 / GB PAYG |
| Oxylabs | 100M+ | Enterprise SLA, 99.95% success | Enterprise tiers |
| Decodo (ex-Smartproxy) | 125M+ | Best price-to-performance | ~$2.25 / GB at volume |
| IPRoyal | 10M+ | Budget, non-expiring bandwidth | ~$1 / GB |
| Webshare | 10M+ | Free tier (10 DC IPs, no CC) | Cheapest ISP |

43% of scraping teams use 2-3 providers for redundancy (Apify 2026 survey).

## Web Unlocker APIs (Managed Bypass)

When build cost exceeds buy cost:

| Provider | Typical price | Success rate | Notes |
|----------|---------------|--------------|-------|
| Bright Data Web Unlocker | ~$3.40 / 1K requests | ~97.9% | Handles proxy + fingerprint + CAPTCHA in one call |
| Oxylabs Web Unblocker | Similar | High | Comparable features |
| ZenRows | Tiered | High | Good API ergonomics |

Decision rule: if your team spends > 2 engineer-weeks / quarter maintaining evasion, switch at least the protected-target portion of the pipeline to Web Unlocker.

## Rate Limiting with Real HTTP Clients

```python
from pyrate_limiter import Duration, Rate, Limiter, RedisBucket
from redis import Redis
from curl_cffi.requests import AsyncSession

limiter = Limiter(RedisBucket.init(
    [Rate(2, Duration.SECOND), Rate(60, Duration.MINUTE), Rate(500, Duration.HOUR)],
    Redis.from_url("redis://localhost"),
    "scraper-bucket",
))

async with AsyncSession() as s:
    async with limiter.async_acquire("api.example.com"):
        r = await s.get(url, impersonate="chrome")
```

Use Redis-backed buckets for distributed scrapers; in-memory buckets are fine for single-process runs.

## Diagnosing "It Works in Browser but 403 in curl_cffi"

Checklist (in order):
1. TLS fingerprint matches browser family? (`impersonate="chrome"` vs Firefox-issued cookies)
2. User-Agent matches the browser that solved the challenge?
3. IP matches (same proxy, same CGNAT block)?
4. `Accept-Language` and `Accept` headers present and plausible?
5. `Referer` / `Origin` set correctly for XHR endpoints?
6. Cookies include `__cf_bm` (bot-management cookie) in addition to `cf_clearance`?
7. TLS profile version -- did Chrome update? (`impersonate="chrome"` auto-tracks latest, pin if you need stability)

## Synergies

- Hand off JS-challenge solving and session warming to `stealth-browser-expert`
- Hand off extraction (LLM / CSS / schema) to `ai-scraping-expert`
- Overall cost/framework/architecture decisions stay with `grabber-architect`