Scraping
Unified skill for web scraping and crawling workflows.
Tier Decision Tree
Single page, quick & free → JinaReader (r.jina.ai)
Single page, anti-bot needed → BrightData (4-tier escalation)
URL discovery, site mapping → Katana (fast crawl, URLs only)
Multi-page, free → Crawl4AI (self-hosted Python)
Multi-page, at scale → Spider.cloud (~$0.48/1K pages)
Multi-page, anti-bot → BrightData Crawl API
Social media platforms → Apify actors
X/Twitter → bird CLI
Discord servers → discrawl (local SQLite archive)
Workflow Routing
| Request Pattern |
Route To |
| Quick scrape, free scrape, jina reader, lightweight extract |
JinaReader/SKILL.md |
| katana, URL discovery, crawl URLs, site map, discover endpoints |
See Katana section below |
| Crawl website free, crawl pages, crawl4ai, map site |
Crawl4AI/SKILL.md |
| Scale scraping, bulk crawl, spider cloud, high volume |
Spider/SKILL.md |
| Bright Data, proxy, anti-bot, CAPTCHA, progressive scraping |
BrightData/SKILL.md |
| read tweet, search X, post tweet, reply tweet, bird, X timeline |
Bird/SKILL.md |
| Discord search, Discord history, Discord messages, Discord members, Discord archive, discrawl |
Discord/SKILL.md |
| Instagram, LinkedIn, TikTok, YouTube, Facebook, Google Maps, Amazon, Apify |
Apify/SKILL.md |
Error Fallback Chain
When a tier fails, escalate to the next tier with anti-bot capabilities:
JinaReader fails (empty/garbage/403) → try Crawl4AI (has Playwright)
Crawl4AI fails (timeout/Playwright) → try Spider.cloud (cloud-based)
Spider.cloud fails (rate limit/block) → try BrightData (proxy rotation)
BrightData fails (all 4 tiers) → report failure, suggest manual approach
For social media: Apify actor fails → check if bird CLI covers the platform (X only). No cross-platform fallback.
For Discord: discrawl is the dedicated tool — no fallback chain. If discrawl fails, check discrawl doctor for diagnostics.
Katana — Fast URL Discovery
Use Katana for mapping a site's URL structure before committing to a full crawl with Spider.cloud or Crawl4AI. Faster than full crawlers when you only need URLs, not content.
# Crawl and discover URLs (depth 2)
katana -u https://example.com -d 2 -o urls.txt
# JavaScript rendering mode
katana -u https://example.com -headless -d 3
# Filter by extension (exclude images/CSS)
katana -u https://example.com -ef png,jpg,gif,css
# JSON output
katana -u https://example.com -jsonl -o results.jsonl
# Scope control (stay on domain)
katana -u https://example.com -fs "example.com" -d 5
When to use: Before Spider.cloud or Crawl4AI, when you need to map URL structure first. Also useful for discovering API endpoints, sitemaps, and hidden pages.
Cost Overview
| Tool |
Cost |
Best For |
| Jina Reader |
Free (1M tokens/mo) |
Single pages, JS-rendered |
| Crawl4AI |
Free (self-hosted) |
Multi-page crawling <500 pages |
| Apify |
Free tier ($5/mo) |
Social media, lead gen |
| bird CLI |
Free |
X/Twitter |
| discrawl |
Free (self-hosted) |
Discord server archiving + search |
| Spider.cloud |
~$0.48/1K pages |
Scale crawling 100-100K pages |
| BrightData |
~$500+/mo |
Anti-bot, Cloudflare bypass |
Examples
Example 1: User: "[typical request]" → Routes to appropriate sub-skill workflow
Example 2: User: "[another request]" → Routes to different sub-skill workflow
1---2name: scraping3description: Web scraping via progressive escalation — Jina Reader, Crawl4AI, Spider.cloud, Bright Data, Apify actors, bird CLI for X/Twitter, discrawl for Discord, Katana URL discovery. USE WHEN scraping, crawl, scrape URL, Twitter scraping, Apify, bot detection, jina reader, crawl4ai, spider cloud, katana, URL discovery, bird, read tweet, search X, Discord search, Discord history, Discord messages, Discord members, Discord archive, discrawl.4---56# Scraping78Unified skill for web scraping and crawling workflows.910## Tier Decision Tree1112```13Single page, quick & free → JinaReader (r.jina.ai)14Single page, anti-bot needed → BrightData (4-tier escalation)15URL discovery, site mapping → Katana (fast crawl, URLs only)16Multi-page, free → Crawl4AI (self-hosted Python)17Multi-page, at scale → Spider.cloud (~$0.48/1K pages)18Multi-page, anti-bot → BrightData Crawl API19Social media platforms → Apify actors20X/Twitter → bird CLI21Discord servers → discrawl (local SQLite archive)22```2324## Workflow Routing2526| Request Pattern | Route To |27|---|---|28| Quick scrape, free scrape, jina reader, lightweight extract | `JinaReader/SKILL.md` |29| katana, URL discovery, crawl URLs, site map, discover endpoints | See **Katana** section below |30| Crawl website free, crawl pages, crawl4ai, map site | `Crawl4AI/SKILL.md` |31| Scale scraping, bulk crawl, spider cloud, high volume | `Spider/SKILL.md` |32| Bright Data, proxy, anti-bot, CAPTCHA, progressive scraping | `BrightData/SKILL.md` |33| read tweet, search X, post tweet, reply tweet, bird, X timeline | `Bird/SKILL.md` |34| Discord search, Discord history, Discord messages, Discord members, Discord archive, discrawl | `Discord/SKILL.md` |35| Instagram, LinkedIn, TikTok, YouTube, Facebook, Google Maps, Amazon, Apify | `Apify/SKILL.md` |3637## Error Fallback Chain3839When a tier fails, escalate to the next tier with anti-bot capabilities:4041```42JinaReader fails (empty/garbage/403) → try Crawl4AI (has Playwright)43Crawl4AI fails (timeout/Playwright) → try Spider.cloud (cloud-based)44Spider.cloud fails (rate limit/block) → try BrightData (proxy rotation)45BrightData fails (all 4 tiers) → report failure, suggest manual approach46```4748For social media: Apify actor fails → check if bird CLI covers the platform (X only). No cross-platform fallback.4950For Discord: discrawl is the dedicated tool — no fallback chain. If discrawl fails, check `discrawl doctor` for diagnostics.5152## Katana — Fast URL Discovery5354Use Katana for mapping a site's URL structure before committing to a full crawl with Spider.cloud or Crawl4AI. Faster than full crawlers when you only need URLs, not content.5556```bash57# Crawl and discover URLs (depth 2)58katana -u https://example.com -d 2 -o urls.txt5960# JavaScript rendering mode61katana -u https://example.com -headless -d 36263# Filter by extension (exclude images/CSS)64katana -u https://example.com -ef png,jpg,gif,css6566# JSON output67katana -u https://example.com -jsonl -o results.jsonl6869# Scope control (stay on domain)70katana -u https://example.com -fs "example.com" -d 571```7273**When to use:** Before Spider.cloud or Crawl4AI, when you need to map URL structure first. Also useful for discovering API endpoints, sitemaps, and hidden pages.7475## Cost Overview7677| Tool | Cost | Best For |78|---|---|---|79| Jina Reader | Free (1M tokens/mo) | Single pages, JS-rendered |80| Crawl4AI | Free (self-hosted) | Multi-page crawling <500 pages |81| Apify | Free tier ($5/mo) | Social media, lead gen |82| bird CLI | Free | X/Twitter |83| discrawl | Free (self-hosted) | Discord server archiving + search |84| Spider.cloud | ~$0.48/1K pages | Scale crawling 100-100K pages |85| BrightData | ~$500+/mo | Anti-bot, Cloudflare bypass |8687## Examples8889**Example 1:** `User: "[typical request]"` → Routes to appropriate sub-skill workflow9091**Example 2:** `User: "[another request]"` → Routes to different sub-skill workflow
mj-deving