octos-org deep_crawl
Overview
The deep_crawl tool recursively crawls a website using a headless Chrome browser via the Chrome DevTools Protocol (CDP). It renders JavaScript, follows same-origin links via BFS, extracts text content from each page, and saves results to disk. This is ideal for crawling JS-rendered SPAs, documentation sites, and any site that requires a full browser environment.
Requirements
- Google Chrome or Chromium must be installed and available in PATH, or at a standard system location.
- macOS:
/Applications/Google Chrome.app/Contents/MacOS/Google Chrome - Linux:
google-chrome,google-chrome-stable, orchromium-browser
- macOS:
Usage
Call the deep_crawl tool with a starting URL. The crawler will follow same-origin links up to the specified depth and page limits.
Parameters
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
url |
string | yes | -- | The seed URL to start crawling from |
max_depth |
integer | no | 3 | Maximum link-following depth (1-10) |
max_pages |
integer | no | 50 | Maximum number of pages to crawl (1-200) |
path_prefix |
string | no | -- | Only follow links whose path starts with this prefix |
Example
{
"url": "https://docs.example.com/guide/",
"max_depth": 3,
"max_pages": 30,
"path_prefix": "/guide/"
}
Output
The tool returns a JSON object on stdout:
{
"output": "# Deep Crawl: https://docs.example.com/guide/\nCrawled 12 pages ...\n\n## Sitemap\n1. [depth=0] https://docs.example.com/guide/ (OK)\n...",
"success": true
}
The output field contains:
- A sitemap listing all crawled pages with their depth and status
- A content preview (first ~2000 characters) for each page
- The directory path where full page contents are saved as
.mdfiles
Results are saved to a research directory named crawl-<hostname>/ under the current working directory. Each page is saved as a numbered markdown file (e.g., 000_index.md, 001_docs_install.md).
Behavior Details
- Only
http://andhttps://URLs are allowed - Only same-origin links are followed (no cross-domain crawling)
- The crawler uses stealth techniques to avoid bot detection (custom user-agent, webdriver flag removal)
- Pages that appear empty or bot-blocked are retried with longer wait times
- URL fragments are stripped and trailing slashes normalized to avoid duplicate visits
- Private/internal IP addresses are blocked (SSRF protection)