Cloudflare Crawl

Use the Cloudflare Browser Rendering /crawl REST API to crawl websites, extract content as HTML/Markdown/JSON, and build knowledge bases. Also covers making websites crawlable — robots.txt, sitemaps, WAF skip rules, semantic HTML, and Cloudflare AI Crawl Control. Use when the user wants to crawl a site, scrape pages, extract structured data, build RAG pipelines via the Cloudflare API, or make their site crawlable by Cloudflare's crawler.

portdeveloper Updated

File contents

portdeveloper/cloudflare-crawl commit 33edfd3906

Frequently asked questions

npx get-skill add portdeveloper/cloudflare-crawl