# Cloudflare Crawl

> Use the Cloudflare Browser Rendering /crawl REST API to crawl websites, extract content as HTML/Markdown/JSON, and build knowledge bases. Also covers making websites crawlable — robots.txt, sitemaps, WAF skip rules, semantic HTML, and Cloudflare AI Crawl Control. Use when the user wants to crawl a site, scrape pages, extract structured data, build RAG pipelines via the Cloudflare API, or make their site crawlable by Cloudflare's crawler.

- Skill: `portdeveloper/cloudflare-crawl` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add portdeveloper/cloudflare-crawl`
- Raw SKILL.md: https://api.skillmd.com/api/skills/portdeveloper/cloudflare-crawl/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- License: MIT
- Author: portdeveloper (https://skillmd.com/u/portdeveloper)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/portdeveloper/cloudflare-crawl

---


# Cloudflare Browser Rendering /crawl Endpoint

Crawl websites and extract content using Cloudflare's Browser Rendering REST API. Supports HTML, Markdown, and structured JSON output. Currently in open beta, available on Workers Free and Paid plans.

This skill covers both sides:
- **Consuming the API** — crawling sites and extracting content (this file)
- **Making sites crawlable** — robots.txt, sitemaps, WAF rules, semantic HTML (see [references/making-sites-crawlable.md](references/making-sites-crawlable.md))

## Workflow

The API is asynchronous with two steps:

1. **POST** to initiate a crawl and receive a job ID
2. **GET** to poll for status and retrieve results

## Authentication

All requests require a Cloudflare API token via `Authorization: Bearer <token>`. You also need your Cloudflare `account_id`.

## Initiate a Crawl

```bash
curl -X POST "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "limit": 10,
    "formats": ["markdown"]
  }'
```

**Response:**

```json
{
  "success": true,
  "result": "c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e"
}
```

The `result` is the job ID.

## Check Results

```bash
curl "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"
```

**Query parameters for results:**

| Parameter | Description |
|-----------|-------------|
| `limit`   | Max records to return |
| `cursor`  | Pagination token (responses paginate at 10 MB) |
| `status`  | Filter by URL status: `queued`, `completed`, `disallowed`, `skipped`, `errored`, `cancelled` |

**Lightweight status poll** (returns minimal data):

```bash
curl "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}?limit=1"
```

## Cancel a Crawl

```bash
curl -X DELETE "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"
```

## Request Parameters

### Required

| Parameter | Type | Description |
|-----------|------|-------------|
| `url` | string | Starting URL to crawl |

### Optional

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `limit` | number | 10 | Max pages to crawl (max: 100,000) |
| `depth` | number | 100,000 | Max link depth from starting URL |
| `source` | string | `"all"` | URL discovery: `"all"`, `"sitemaps"`, or `"links"` |
| `formats` | array | `["html"]` | Output formats: `"html"`, `"markdown"`, `"json"` |
| `render` | boolean | `true` | `true` = headless browser, `false` = fast static HTML fetch |
| `maxAge` | number | 86400 | Cache duration in seconds (max: 604,800) |
| `modifiedSince` | number | - | Unix timestamp; skip pages unchanged since this time |
| `options.includeExternalLinks` | boolean | `false` | Follow links to external domains |
| `options.includeSubdomains` | boolean | `false` | Follow links to subdomains |
| `options.includePatterns` | array | - | Wildcard patterns for URLs to include |
| `options.excludePatterns` | array | - | Wildcard patterns for URLs to exclude |
| `jsonOptions` | object | - | AI extraction config: `prompt`, `response_format`, `custom_ai` |
| `authenticate` | object | - | HTTP basic auth: `{ "username": "...", "password": "..." }` |
| `setExtraHTTPHeaders` | object | - | Custom headers sent with each request |
| `userAgent` | string | - | Custom user agent |
| `gotoOptions` | object | - | Navigation: `waitUntil`, `timeout` |
| `waitForSelector` | object | - | Wait for element: `selector`, `timeout`, `visible` |
| `rejectResourceTypes` | array | - | Block: `"image"`, `"media"`, `"font"`, `"stylesheet"` |

## Pattern Matching

- `*` matches any characters except `/`
- `**` matches any characters including `/`
- `excludePatterns` always takes priority over `includePatterns`

## Response Structure

```json
{
  "success": true,
  "result": {
    "id": "job-id",
    "status": "completed",
    "browserSecondsUsed": 134.7,
    "total": 50,
    "finished": 50,
    "records": [
      {
        "url": "https://example.com/page",
        "status": "completed",
        "markdown": "# Page Content...",
        "html": "<html>...</html>",
        "json": {},
        "metadata": {
          "status": 200,
          "title": "Page Title",
          "url": "https://example.com/page"
        }
      }
    ],
    "cursor": "next-page-token"
  }
}
```

## Job Statuses

| Status | Meaning |
|--------|---------|
| `running` | Crawl in progress |
| `completed` | Finished successfully |
| `cancelled_due_to_timeout` | Exceeded 7-day limit |
| `cancelled_due_to_limits` | Hit account limits |
| `cancelled_by_user` | Manually cancelled |
| `errored` | Encountered an error |

## Limits

- Jobs run for up to **7 days**; results retained for **14 days**
- Free plan: **10 minutes** browser time per day
- `render: true` uses headless browser (billed as browser rendering time)
- `render: false` uses Workers (currently unbilled in beta)

## Examples

### Crawl a docs site for markdown

```json
{
  "url": "https://docs.example.com",
  "limit": 100,
  "formats": ["markdown"],
  "source": "sitemaps",
  "options": {
    "includePatterns": ["https://docs.example.com/guide/**"]
  }
}
```

### Fast static crawl (no JS rendering)

```json
{
  "url": "https://blog.example.com",
  "limit": 50,
  "render": false,
  "formats": ["markdown"],
  "rejectResourceTypes": ["image", "media", "font", "stylesheet"]
}
```

### Extract structured JSON with AI

```json
{
  "url": "https://shop.example.com/products",
  "limit": 20,
  "formats": ["json"],
  "jsonOptions": {
    "prompt": "Extract the product name, price, and description from this page.",
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "product",
        "schema": {
          "type": "object",
          "properties": {
            "name": { "type": "string" },
            "price": { "type": "string" },
            "description": { "type": "string" }
          }
        }
      }
    }
  }
}
```

### Incremental crawl (only new/changed pages)

```json
{
  "url": "https://news.example.com",
  "limit": 200,
  "modifiedSince": 1741564800,
  "maxAge": 3600
}
```

## Troubleshooting

- **Empty results**: Check `robots.txt` allows crawling. Try `source: "sitemaps"` or increase `depth`. Verify `includePatterns` match actual URLs.
- **Slow crawls**: Use `render: false` for static content. Block unnecessary resources with `rejectResourceTypes`. Run multiple smaller crawls.
- **Cancelled due to limits**: Use `render: false`, increase `maxAge` for caching, reduce `limit`, or upgrade to Workers Paid plan.
- **JSON extraction issues**: Write detailed extraction prompts, define a response schema, or specify a custom AI model.

## Making Your Site Crawlable

If the user is a site owner wanting their content to be crawlable, see [references/making-sites-crawlable.md](references/making-sites-crawlable.md) for detailed guidance on:

- **robots.txt** — allowing the crawler and setting crawl-delay
- **XML sitemaps** — structure and `<lastmod>` for incremental crawls
- **WAF skip rules** — bypassing Bot Management/WAF/Turnstile for the crawler
- **Cloudflare AI Crawl Control** — monitoring and managing AI crawler access
- **Semantic HTML** — clean structure for better markdown/JSON extraction
- **JSON extraction optimization** — consistent layouts and clear labeling

## Important Notes

- The crawler respects `robots.txt` and `crawl-delay` directives. Blocked URLs appear with status `"disallowed"`.
- Bot protection (Bot Management, WAF, Turnstile) on target sites applies to the crawler.
- URL discovery order when `source: "all"`: starting URL, then sitemap URLs, then page links.