# Deep Crawl

> deep_crawl

- Skill: `zhanghandong/deep-crawl` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add zhanghandong/deep-crawl`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zhanghandong/deep-crawl/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ZhangHanDong (https://skillmd.com/u/zhanghandong)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zhanghandong/deep-crawl

---


# deep_crawl

## Overview

The `deep_crawl` tool recursively crawls a website using a headless Chrome browser via the Chrome DevTools Protocol (CDP). It renders JavaScript, follows same-origin links via BFS, extracts text content from each page, and saves results to disk. This is ideal for crawling JS-rendered SPAs, documentation sites, and any site that requires a full browser environment.

## Requirements

- **Google Chrome** or **Chromium** must be installed and available in PATH, or at a standard system location.
  - macOS: `/Applications/Google Chrome.app/Contents/MacOS/Google Chrome`
  - Linux: `google-chrome`, `google-chrome-stable`, or `chromium-browser`

## Usage

Call the `deep_crawl` tool with a starting URL. The crawler will follow same-origin links up to the specified depth and page limits.

### Parameters

| Parameter     | Type    | Required | Default | Description                                              |
|---------------|---------|----------|---------|----------------------------------------------------------|
| `url`         | string  | yes      | --      | The seed URL to start crawling from                      |
| `max_depth`   | integer | no       | 3       | Maximum link-following depth (1-10)                      |
| `max_pages`   | integer | no       | 50      | Maximum number of pages to crawl (1-200)                 |
| `path_prefix` | string  | no       | --      | Only follow links whose path starts with this prefix     |

### Example

```json
{
  "url": "https://docs.example.com/guide/",
  "max_depth": 3,
  "max_pages": 30,
  "path_prefix": "/guide/"
}
```

## Output

The tool returns a JSON object on stdout:

```json
{
  "output": "# Deep Crawl: https://docs.example.com/guide/\nCrawled 12 pages ...\n\n## Sitemap\n1. [depth=0] https://docs.example.com/guide/ (OK)\n...",
  "success": true
}
```

The `output` field contains:
- A **sitemap** listing all crawled pages with their depth and status
- A **content preview** (first ~2000 characters) for each page
- The **directory path** where full page contents are saved as `.md` files

Results are saved to a research directory named `crawl-<hostname>/` under the current working directory. Each page is saved as a numbered markdown file (e.g., `000_index.md`, `001_docs_install.md`).

## Behavior Details

- Only `http://` and `https://` URLs are allowed
- Only same-origin links are followed (no cross-domain crawling)
- The crawler uses stealth techniques to avoid bot detection (custom user-agent, webdriver flag removal)
- Pages that appear empty or bot-blocked are retried with longer wait times
- URL fragments are stripped and trailing slashes normalized to avoid duplicate visits
- Private/internal IP addresses are blocked (SSRF protection)