# Grabber Development AI Scraping Expert

> Specialist for the AI-assisted half of a scraping pipeline. TRIGGER WHEN: picking between Crawl4AI, Firecrawl, ScrapeGraphAI, Browser Use, Stagehand, Skyvern, Jina Reader and Spider.cloud, designing Pydantic schema-driven extraction, building a CSS plus LLM-fallback hybrid, reverse-engineering a GraphQL API, or estimating extraction cost at scale. DO NOT TRIGGER WHEN: the work is TLS fingerprinting (use http-fingerprint-expert), browser stealth or CAPTCHA (use stealth-browser-expert), or boilerplate with no LLM (use grabber-architect).

- Skill: `acaprino/grabber-development-ai-scraping-expert` (Agent Skill)
- Install (CLI): `npx skillmds@latest add acaprino/grabber-development-ai-scraping-expert`
- Raw SKILL.md: https://api.skillmd.com/api/skills/acaprino/grabber-development-ai-scraping-expert/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: acaprino (https://skillmd.com/u/acaprino)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/acaprino/grabber-development-ai-scraping-expert

---


<!-- Generated by the Daodan compiler for pi. Edit the kernel, never this file. -->

# AI-Assisted Scraping Expert

Expertise in LLM-assisted scraping frameworks and extraction strategies. You pick the right framework for the data shape, design schema-driven extraction with LLM fallback, and keep cost-per-record under control. You do NOT design evasion -- that is `stealth-browser-expert` / `http-fingerprint-expert`.

## Framework Matrix

| Framework | Stars (2026) | Core model | Best for | Cost |
|-----------|--------------|-----------|----------|------|
| Crawl4AI (v0.8+) | ~51K | LLM-friendly markdown, deep crawl | Converting sites to LLM-ready corpora | Self-hosted + LLM |
| Firecrawl | Enterprise OSS | Site -> markdown, FIRE-1 agent | Schema-driven extraction at scale | ~$19-2500/mo tiers |
| ScrapeGraphAI (v1.74+) | ~18K | Graph-based LLM pipelines | Natural-language extraction | Self-hosted + LLM |
| Browser Use (v0.12+) | ~85K | Playwright + any LLM | Autonomous browsing agents (89% WebVoyager) | Self-hosted + LLM |
| Stagehand | ~21K | Auto-caching + self-healing | Stable automation with LLM-only on DOM change | Self-hosted + LLM |
| Skyvern | ~20K | Vision-first | Understanding pages via screenshots (forms, CAPTCHAs) | Self-hosted + vision LLM |
| Jina Reader | SaaS | 100B tokens/day, ReaderLM-v2 | URL -> clean markdown in 1 HTTP call | `https://r.jina.ai/` prefix |
| Spider.cloud | SaaS | Rust engine, multimodal | Bulk crawl + LLM in one service | ~$0.48 / 1K pages |

## When to Use LLM Extraction vs CSS

| Data location | Method | Cost/record | Stability |
|---------------|--------|-------------|-----------|
| API endpoint (REST/GraphQL) | Replay with `curl_cffi` | $0 | Highest -- changes with API |
| JSON-LD / microdata | Parse inline JSON | $0 | High -- schema.org standards |
| Inline `<script>` JSON | Regex / `json.loads` | $0 | High -- rarely changes |
| Stable DOM (e-commerce cards) | CSS / XPath selectors | $0 | Medium -- brittle to redesign |
| Unstable DOM | LLM extraction | ~$0.01 | High -- survives DOM changes |
| Mixed | CSS primary + LLM fallback | ~$0.001 avg | High |

Rule of thumb: always try API interception first (phase 1), inline JSON second (phase 2), LLM last (phase 3). LLMs are a maintenance reducer, not a first choice.

## Hybrid CSS + LLM Pattern (Recommended Default)

```python
from pydantic import BaseModel
from typing import Optional
import json
from selectolax.parser import HTMLParser

class Product(BaseModel):
    title: str
    price_usd: float
    sku: Optional[str] = None
    in_stock: bool

async def extract_product(html: str, llm_client) -> Product:
    # Stage 1: try CSS selectors (fast, free)
    tree = HTMLParser(html)
    try:
        return Product(
            title=tree.css_first("h1.product-title").text(strip=True),
            price_usd=float(tree.css_first("span.price").text(strip=True).replace("$", "")),
            sku=tree.css_first("meta[itemprop='sku']").attributes.get("content"),
            in_stock=tree.css_first("link[itemprop='availability']").attributes.get("href", "").endswith("InStock"),
        )
    except (AttributeError, ValueError):
        pass

    # Stage 2: JSON-LD fallback (free)
    for script in tree.css('script[type="application/ld+json"]'):
        try:
            data = json.loads(script.text())
            if data.get("@type") == "Product":
                return Product(**map_jsonld_to_product(data))
        except (json.JSONDecodeError, TypeError):
            continue

    # Stage 3: LLM extraction (~$0.01, slowest)
    return await llm_client.extract(html, schema=Product)
```

This pattern keeps cost near zero for 90% of pages and only pays for LLM on redesigns or one-off outliers.

## Pydantic Schema-Driven Extraction

Define the output model once, reuse across all extraction strategies:

```python
from pydantic import BaseModel, HttpUrl, Field
from typing import Optional
from datetime import datetime

class JobListing(BaseModel):
    title: str
    company: str
    location: Optional[str] = None
    remote: bool = False
    salary_min: Optional[int] = Field(None, ge=0)
    salary_max: Optional[int] = Field(None, ge=0)
    salary_currency: Optional[str] = Field(None, pattern="^[A-Z]{3}$")
    posted_at: Optional[datetime] = None
    apply_url: HttpUrl

# Firecrawl
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key=KEY)
result = app.scrape_url(url, params={"formats": ["extract"], "extract": {"schema": JobListing.model_json_schema()}})
job = JobListing(**result["extract"])

# Crawl4AI
from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy, LLMExtractionStrategy

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(url, extraction_strategy=LLMExtractionStrategy(
        provider="openai/gpt-4o-mini",
        api_token=KEY,
        schema=JobListing.model_json_schema(),
        extraction_type="schema",
    ))
    job = JobListing.model_validate_json(result.extracted_content)
```

## Framework Picks by Use Case

### "Convert a site to LLM-ready markdown"
-> **Jina Reader** (SaaS, fastest) or **Firecrawl** (self-hostable or cloud)

### "Schema-driven extraction across many pages"
-> **Firecrawl** (strongest Pydantic integration) or **Crawl4AI** (self-hosted)

### "Autonomous agent that navigates and fills forms"
-> **Browser Use** (best WebVoyager score) or **Skyvern** (when you need vision for forms)

### "Self-healing scraper that stays cheap"
-> **Stagehand** -- caches selectors, invokes LLM only when DOM changes; or **Scrapling** for selector auto-relocation

### "Natural-language data pipelines"
-> **ScrapeGraphAI** -- describe the extraction as a graph of LLM + parser nodes

### "High-volume cheap extraction"
-> CSS + LLM hybrid (above) -- pure SaaS is $$$ at 1M+ pages/month

## GraphQL Reverse Engineering

GraphQL endpoints are often the cleanest replay targets.

### Discovery
- InQL v6.1+: Kotlin rewrite, schema brute-forcer + engine fingerprinter + scanner
- `clairvoyance`: reconstruct schema when introspection is disabled, using "did you mean" suggestion abuse
- Check `/graphql`, `/api/graphql`, `/v1/graphql` endpoints
- Check WebSocket frames for GraphQL subscriptions

### Persisted Query Bypass
Many modern APIs use persisted queries (client sends only a sha256Hash). To force full query retransmission:

```python
# mitmproxy addon -- replace sha256Hash with bogus value, forces PersistedQueryNotFound,
# client retransmits the full query which you capture.
import json

def request(flow):
    try:
        data = json.loads(flow.request.text)
        if isinstance(data, list):
            for item in data:
                item["extensions"]["persistedQuery"]["sha256Hash"] = "0000"
        else:
            data["extensions"]["persistedQuery"]["sha256Hash"] = "0000"
        flow.request.text = json.dumps(data)
    except (json.JSONDecodeError, KeyError, TypeError):
        pass
```

### Introspection Bypass Techniques
When `__schema` is blocked:
- Whitespace variations (`__schema` vs `__schema ` with trailing space)
- GET instead of POST
- Inline fragment attacks to leak field types

## Cost Modelling

Per-record cost components:
- **Proxy bandwidth**: ~$0.002 / page (residential at $8/GB, 250KB avg)
- **LLM extraction**: ~$0.01 / page (gpt-4o-mini, 5K input tokens + 500 output)
- **Vision LLM** (Skyvern-style): ~$0.03-0.10 / page (screenshot tokens are expensive)
- **CAPTCHA**: $0.80-2.99 / 1K if present
- **Managed API** (Firecrawl, Spider, Web Unlocker): $0.001-0.01 / page depending on tier

Budget formula:
```
monthly_cost = pages * (proxy_gb_avg * proxy_price + llm_rate + captcha_rate * captcha_price)
```

A 1M page/month pipeline with residential proxies and LLM extraction is ~$10-15K/month. Hybrid CSS+LLM drops that 5-10x.

## Anti-Patterns

- **LLM on every page**: burns budget; always try CSS/JSON-LD first
- **No schema validation**: LLMs hallucinate; Pydantic rejects nonsense
- **Vision LLM as default**: 3-10x cost of text-only; use only for visual forms or CAPTCHAs
- **Single-stage extraction**: no fallback means one redesign kills the pipeline

## Synergies

- Browser for rendering + JS challenges -> `stealth-browser-expert`
- HTTP replay at scale -> `http-fingerprint-expert`
- Pipeline orchestration + cost tracking + framework choice -> `grabber-architect`
- Async pipelines in Python -> `python-development:async-python-patterns`
- Tracing scraper flows -> `opentelemetry:opentelemetry`