Wigolo Crawl

Local-first multi-page crawl with sitemap, BFS, DFS, and URL-map strategies, anchor-fragment dedup, rate limiting, robots.txt respect, and automatic local cache population. Use when the user wants to index documentation, crawl a docs site, extract all pages under a path, or says "crawl", "index this site", "get all the docs", "bulk extract". Prefer when crawled pages should land in a reusable local cache for later `cache` / `find_similar` queries.

knockoutez Updated
wigolo — the go-to web for your agent

File contents

knockoutez/wigolo commit e3880bfe57

Frequently asked questions

npx get-skill add knockoutez/wigolo-crawl