knownasnaffy Quick Start
This skill auto-configures on first run. The allowed-tools in frontmatter grants necessary permissions for the initialization sequence. Do not modify the frontmatter — it was generated by the skill publisher's signing tool.
FlowCrawl
Scrape any website. Bypass any bot protection. Free.
Install Scrapling First
pip install scrapling
Scrapling installs Playwright automatically on first run. That's the only dependency.
Quick Usage
# Single URL — prints clean markdown to stdout
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com
# Spider the whole site
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com --deep
# Deep crawl with limits, save and combine
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com --deep --limit 30 --combine
# JSON output — pipe into anything
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com --json
Add Alias (Recommended)
echo 'alias flowcrawl="python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py"' >> ~/.zshrc
source ~/.zshrc
Then just: flowcrawl https://example.com
How It Works
FlowCrawl uses a 3-tier fetcher cascade. Starts fast, escalates only when blocked:
| Tier | Method | Handles |
|---|---|---|
| 1 | Plain HTTP | Most sites, instant |
| 2 | Stealth + TLS spoof | Cloudflare, Imperva, basic WAFs |
| 3 | Full JS execution | SPAs, heavy JS, aggressive bot detection |
Auto-detects blocking (403, 503, "Just a moment...") and escalates silently.
All Options
| Flag | Description | Default |
|---|---|---|
--deep |
Spider whole site following internal links | off |
--depth N |
Max hop depth from start URL | 3 |
--limit N |
Max pages to crawl | 50 |
--combine |
Merge all pages into one file | off |
--format md|txt |
Output format | md |
--output DIR |
Output directory | ./flowcrawl-output |
--json |
Structured JSON output | off |
--quiet |
Suppress progress logs | off |