firecrawl-policy-guardrails
Skill Scaffold
firecrawl-policy-guardrails/
SKILL.md
File Descriptions
1. SKILL.md
Purpose: Implement robots.txt compliance, rate limiting policies, content filtering, and ethical scraping guardrails. Workflow: Compliance skill - ensures scraping operations are ethical, legal, and respect target site policies. Relates to: Extends firecrawl-security-basics; supports firecrawl-data-handling for responsible data use
Summary
This skill implements ethical and legal guardrails for web scraping. It covers robots.txt parsing and compliance, respecting crawl-delay directives, domain blocklists for protected content, rate limiting to avoid site disruption, noindex/nofollow respect, terms of service awareness, and content licensing considerations. These guardrails ensure scraping operations are responsible, legal, and maintain good standing with content providers.