majiayu000 Scrapling -- Local Stealth Web Scraping
100% local Python library (BSD-3, D4Vinci/Scrapling). No API keys, no cloud dependencies. Built-in Cloudflare solver, TLS impersonation, and adaptive element tracking.
When to Use Scrapling vs Firecrawl
| Need | Use | Why |
|---|---|---|
| Clean markdown from a URL | firecrawl scrape --only-main-content |
Optimized for LLM markdown conversion |
| Bypass Cloudflare/anti-bot | scrapling stealth fetch |
Built-in Turnstile solver, Patchright stealth |
| Extract specific elements | scrapling with CSS/XPath selectors |
Element-level precision, adaptive tracking |
| No API key available | scrapling |
100% local, zero credentials |
| Batch cloud scraping | firecrawl crawl / batch-scrape |
Cloud infrastructure, parallel processing |
| Site redesign resilience | scrapling adaptive mode |
SQLite-backed similarity matching |
| Full-site concurrent crawl | scrapling Spider framework |
Scrapy-like with pause/resume |
| Web search + scrape | firecrawl search --scrape |
Combined search + extraction |
Installation
Run once to set up Scrapling with all features:
~/.claude/skills/scrapling/scripts/scrapling_install.sh
Installs scrapling[all] via uv, downloads Chromium + system dependencies, and verifies all fetchers load.
Quick Start -- Stdout Wrapper
The wrapper uses Scrapling's Python API directly (faster than CLI, avoids curl_cffi cert issues) and outputs to stdout for piping into filter_web_results.py:
# Basic HTTP fetch (fastest, TLS impersonation)
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://example.com
# Stealth mode (Patchright, anti-bot bypass)
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://protected.site --stealth
# Stealth + Cloudflare solver
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://cf-protected.site --stealth --solve-cloudflare
# Dynamic (Playwright Chromium, JS rendering)
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://js-heavy.site --dynamic
# With CSS selector for targeted extraction
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://example.com --css ".product-list"
# Pipe through firecrawl's filter for token efficiency
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://example.com --stealth | \
python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --sections "Pricing" --max-chars 5000
Full path: python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py
Flags: --stealth, --dynamic, --css SELECTOR, --solve-cloudflare, --impersonate BROWSER, --format {text,html}, --no-headless, --timeout SECONDS
For --network-idle, --real-chrome, or POST requests, use the CLI direct path below.
Quick Start -- CLI Direct
For file-based output (Scrapling's native CLI):
# HTTP fetch -> markdown
scrapling extract get 'https://example.com' content.md
# HTTP with CSS selector and browser impersonation
scrapling extract get 'https://example.com' content.md --css-selector '.main-content' --impersonate chrome
# Dynamic fetch (Playwright, JS rendering)
scrapling extract fetch 'https://example.com' content.md
# Stealth with Cloudflare bypass
scrapling extract stealthy-fetch 'https://protected.site' content.md --solve-cloudflare
# Stealth with CSS selector, visible browser for debugging
scrapling extract stealthy-fetch 'https://site.com' content.md --css-selector '.data' --no-headless
Three Fetcher Tiers
| CLI verb | Engine | Stealth | JS | Speed |
|---|---|---|---|---|
get |
curl_cffi (HTTP) | TLS impersonation | No | Fast |
fetch |
Playwright/Chromium | Medium | Yes | Medium |
stealthy-fetch |
Patchright/Chrome | Maximum | Yes | Slower |
Python API
For element-level extraction, automation, or when the CLI is insufficient:
from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher
# Simple HTTP fetch with CSS extraction
page = Fetcher.get('https://example.com', impersonate='chrome')
titles = page.css('.item h2::text').getall()
links = page.css('a::attr(href)').getall()
# Stealth with Cloudflare bypass
page = StealthyFetcher.fetch('https://protected.site',
headless=True, solve_cloudflare=True,
hide_canvas=True, block_webrtc=True)
data = page.css('.content').get_all_text()
# Page automation (login, click, fill)
def login(page):
page.fill('#username', 'user')
page.fill('#password', 'pass')
page.click('#submit')
page = StealthyFetcher.fetch('https://app.example.com', page_action=login)
Parsing (no fetching)
from scrapling.parser import Selector
page = Selector("<html>...</html>")
page.css('.item::text').getall() # CSS with pseudo-elements
page.xpath('//div[@class="item"]') # XPath
page.find_all('div', class_='item') # BeautifulSoup-style
page.find_by_text('Add to Cart') # Text search
page.find_by_regex(r'Price: \$\d+') # Regex search
Adaptive Scraping
Scrapling's signature feature. Elements are fingerprinted to SQLite and relocated by similarity scoring after site redesigns.
from scrapling.fetchers import Fetcher
Fetcher.adaptive = True
page = Fetcher.get('https://example.com')
# First run: save element fingerprint
products = page.css('.product-list', auto_save=True)
# Later, after site redesign breaks the selector:
products = page.css('.product-list', adaptive=True) # Still finds it
Read: references/adaptive-scraping.md
Spider Framework
For full-site crawling with concurrency, pause/resume, and session routing:
from scrapling.spiders import Spider, Response, Request
from scrapling.fetchers import FetcherSession, StealthySession
class ResearchSpider(Spider):
name = "research"
start_urls = ["https://example.com/"]
concurrent_requests = 10
def configure_sessions(self, manager):
manager.add("fast", FetcherSession(impersonate="chrome"))
manager.add("stealth", StealthySession(headless=True), lazy=True)
async def parse(self, response: Response):
for link in response.css('a::attr(href)').getall():
if "protected" in link:
yield Request(link, sid="stealth")
else:
yield Request(link, sid="fast")
for item in response.css('.article'):
yield {"title": item.css('h2::text').get(), "url": response.url}
result = ResearchSpider().start(crawldir="./data") # Pause/resume enabled
result.items.to_json("output.json")
Troubleshooting
- Browser not found: Run
scrapling installto download browser dependencies - Import error on fetchers: Install the full package:
uv pip install "scrapling[all]" - Cloudflare still blocking: Combine flags:
--solve-cloudflare --block-webrtc --hide-canvas - SSL cert error with HTTP fetcher: curl_cffi's bundled CA certs can be stale on macOS/pyenv. The wrapper handles this automatically (
verify=False). For Python API, passverify=FalsetoFetcher.get(). - Slow stealthy fetch: Expected--browser automation is inherently slower than HTTP requests. Use
get(HTTP) when stealth is not needed.
Reference Documentation
| File | Contents |
|---|---|
references/cli-reference.md |
Full CLI extract command reference (get, post, fetch, stealthy-fetch) |
references/python-api-reference.md |
Python API (Fetcher, DynamicFetcher, StealthyFetcher, Sessions, Spiders) |
references/adaptive-scraping.md |
Adaptive element tracking deep-dive (save, match, similarity scoring) |