Skill Scaffold

This skill implements full website crawling with FireCrawl's crawl endpoint. It covers crawl configuration including max depth, URL include/exclude patterns, sitemap utilization, rate limiting, and page limits.

tools-only Updated 7 repo stars

File contents

firecrawl-crawl-site

Skill Scaffold

firecrawl-crawl-site/
  SKILL.md

File Descriptions

1. SKILL.md

Purpose: Crawl entire websites or sections with depth control, URL filtering, sitemap respect, and comprehensive coverage options. Workflow: Multi-page crawling workflow - extracts content from entire sites or specific sections with intelligent navigation. Relates to: Extends firecrawl-scrape-single; complements firecrawl-batch-processing for high-volume operations

Summary

This skill implements full website crawling with FireCrawl's crawl endpoint. It covers crawl configuration including max depth, URL include/exclude patterns, sitemap utilization, rate limiting, and page limits. The skill handles the asynchronous nature of crawl jobs, polling for completion, and aggregating results. This enables comprehensive content extraction from documentation sites, blogs, knowledge bases, and entire domains.

tools-only/X-Skills/tree/main/automation/scripting/567-firecrawl-crawl-site_4a1ed183 commit 9d0d13c7a0

Frequently asked questions

npx skillmds@latest add tools-only/skill-scaffold-55