Crw Crawl

Crawl an entire website or section and extract content from every page. Use when you need content from many pages under a common URL prefix: "crawl the whole site", "get all docs pages", "scrape every blog post", "download the full docs for RAG", "extract all pages under /api". Async BFS — starts a job and polls for results. Step 4 of the crw workflow ladder.

us Updated 1.3k repo stars
fastCRW compared with Crawl4AI and Firecrawl on truth-recall, unique recoveries, median latency, download size, and recall depth

File contents

us/crw commit 8923573df5

Frequently asked questions

npx get-skill add us/crw-crawl