portdeveloper Cloudflare Browser Rendering /crawl Endpoint
Crawl websites and extract content using Cloudflare's Browser Rendering REST API. Supports HTML, Markdown, and structured JSON output. Currently in open beta, available on Workers Free and Paid plans.
This skill covers both sides:
- Consuming the API — crawling sites and extracting content (this file)
- Making sites crawlable — robots.txt, sitemaps, WAF rules, semantic HTML (see references/making-sites-crawlable.md)
Workflow
The API is asynchronous with two steps:
- POST to initiate a crawl and receive a job ID
- GET to poll for status and retrieve results
Authentication
All requests require a Cloudflare API token via Authorization: Bearer <token>. You also need your Cloudflare account_id.
Initiate a Crawl
curl -X POST "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl" \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"limit": 10,
"formats": ["markdown"]
}'
Response:
{
"success": true,
"result": "c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e"
}
The result is the job ID.
Check Results
curl "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}" \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"
Query parameters for results:
| Parameter | Description |
|---|---|
limit |
Max records to return |
cursor |
Pagination token (responses paginate at 10 MB) |
status |
Filter by URL status: queued, completed, disallowed, skipped, errored, cancelled |
Lightweight status poll (returns minimal data):
curl "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}?limit=1"
Cancel a Crawl
curl -X DELETE "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}" \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"
Request Parameters
Required
| Parameter | Type | Description |
|---|---|---|
url |
string | Starting URL to crawl |
Optional
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
number | 10 | Max pages to crawl (max: 100,000) |
depth |
number | 100,000 | Max link depth from starting URL |
source |
string | "all" |
URL discovery: "all", "sitemaps", or "links" |
formats |
array | ["html"] |
Output formats: "html", "markdown", "json" |
render |
boolean | true |
true = headless browser, false = fast static HTML fetch |
maxAge |
number | 86400 | Cache duration in seconds (max: 604,800) |
modifiedSince |
number | - | Unix timestamp; skip pages unchanged since this time |
options.includeExternalLinks |
boolean | false |
Follow links to external domains |
options.includeSubdomains |
boolean | false |
Follow links to subdomains |
options.includePatterns |
array | - | Wildcard patterns for URLs to include |
options.excludePatterns |
array | - | Wildcard patterns for URLs to exclude |
jsonOptions |
object | - | AI extraction config: prompt, response_format, custom_ai |
authenticate |
object | - | HTTP basic auth: { "username": "...", "password": "..." } |
setExtraHTTPHeaders |
object | - | Custom headers sent with each request |
userAgent |
string | - | Custom user agent |
gotoOptions |
object | - | Navigation: waitUntil, timeout |
waitForSelector |
object | - | Wait for element: selector, timeout, visible |
rejectResourceTypes |
array | - | Block: "image", "media", "font", "stylesheet" |
Pattern Matching
*matches any characters except/**matches any characters including/excludePatternsalways takes priority overincludePatterns
Response Structure
{
"success": true,
"result": {
"id": "job-id",
"status": "completed",
"browserSecondsUsed": 134.7,
"total": 50,
"finished": 50,
"records": [
{
"url": "https://example.com/page",
"status": "completed",
"markdown": "# Page Content...",
"html": "<html>...</html>",
"json": {},
"metadata": {
"status": 200,
"title": "Page Title",
"url": "https://example.com/page"
}
}
],
"cursor": "next-page-token"
}
}
Job Statuses
| Status | Meaning |
|---|---|
running |
Crawl in progress |
completed |
Finished successfully |
cancelled_due_to_timeout |
Exceeded 7-day limit |
cancelled_due_to_limits |
Hit account limits |
cancelled_by_user |
Manually cancelled |
errored |
Encountered an error |
Limits
- Jobs run for up to 7 days; results retained for 14 days
- Free plan: 10 minutes browser time per day
render: trueuses headless browser (billed as browser rendering time)render: falseuses Workers (currently unbilled in beta)
Examples
Crawl a docs site for markdown
{
"url": "https://docs.example.com",
"limit": 100,
"formats": ["markdown"],
"source": "sitemaps",
"options": {
"includePatterns": ["https://docs.example.com/guide/**"]
}
}
Fast static crawl (no JS rendering)
{
"url": "https://blog.example.com",
"limit": 50,
"render": false,
"formats": ["markdown"],
"rejectResourceTypes": ["image", "media", "font", "stylesheet"]
}
Extract structured JSON with AI
{
"url": "https://shop.example.com/products",
"limit": 20,
"formats": ["json"],
"jsonOptions": {
"prompt": "Extract the product name, price, and description from this page.",
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "product",
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "string" },
"description": { "type": "string" }
}
}
}
}
}
}
Incremental crawl (only new/changed pages)
{
"url": "https://news.example.com",
"limit": 200,
"modifiedSince": 1741564800,
"maxAge": 3600
}
Troubleshooting
- Empty results: Check
robots.txtallows crawling. Trysource: "sitemaps"or increasedepth. VerifyincludePatternsmatch actual URLs. - Slow crawls: Use
render: falsefor static content. Block unnecessary resources withrejectResourceTypes. Run multiple smaller crawls. - Cancelled due to limits: Use
render: false, increasemaxAgefor caching, reducelimit, or upgrade to Workers Paid plan. - JSON extraction issues: Write detailed extraction prompts, define a response schema, or specify a custom AI model.
Making Your Site Crawlable
If the user is a site owner wanting their content to be crawlable, see references/making-sites-crawlable.md for detailed guidance on:
- robots.txt — allowing the crawler and setting crawl-delay
- XML sitemaps — structure and
<lastmod>for incremental crawls - WAF skip rules — bypassing Bot Management/WAF/Turnstile for the crawler
- Cloudflare AI Crawl Control — monitoring and managing AI crawler access
- Semantic HTML — clean structure for better markdown/JSON extraction
- JSON extraction optimization — consistent layouts and clear labeling
Important Notes
- The crawler respects
robots.txtandcrawl-delaydirectives. Blocked URLs appear with status"disallowed". - Bot protection (Bot Management, WAF, Turnstile) on target sites applies to the crawler.
- URL discovery order when
source: "all": starting URL, then sitemap URLs, then page links.