434 Firecrawl Policy Guardrails 4a48f641

firecrawl-policy-guardrails

tools-only Updated 7 repo stars

File contents

firecrawl-policy-guardrails

Skill Scaffold

firecrawl-policy-guardrails/
  SKILL.md

File Descriptions

1. SKILL.md

Purpose: Implement robots.txt compliance, rate limiting policies, content filtering, and ethical scraping guardrails. Workflow: Compliance skill - ensures scraping operations are ethical, legal, and respect target site policies. Relates to: Extends firecrawl-security-basics; supports firecrawl-data-handling for responsible data use

Summary

This skill implements ethical and legal guardrails for web scraping. It covers robots.txt parsing and compliance, respecting crawl-delay directives, domain blocklists for protected content, rate limiting to avoid site disruption, noindex/nofollow respect, terms of service awareness, and content licensing considerations. These guardrails ensure scraping operations are responsible, legal, and maintain good standing with content providers.

tools-only/X-Skills/tree/main/automation/workflow/434-firecrawl-policy-guardrails_4a48f641 commit 84db695283

Frequently asked questions

npx skillmds@latest add tools-only/434-firecrawl-policy-guardrails-4a48f641