# Tavily MCP Crawl

> Crawl websites and extract content from multiple pages through the local Tavily MCP server. Use this skill when the user wants to crawl a site, inspect or download documentation, extract an entire docs section, bulk-extract pages, get all pages under a path, or says "crawl", "get all the pages", "download the docs", "extract everything under /docs", or "bulk extract". Supports depth and breadth limits, path/domain filtering, semantic instructions, and markdown or text extraction.

- Skill: `minhngoc25a/tavily-mcp-crawl` (Agent Skill)
- Install (CLI): `npx skillmds@latest add minhngoc25a/tavily-mcp-crawl`
- Raw SKILL.md: https://api.skillmd.com/api/skills/minhngoc25a/tavily-mcp-crawl/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: minhngoc25a (https://skillmd.com/u/minhngoc25a)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/minhngoc25a/tavily-mcp-crawl

---


# Tavily MCP crawl

Use `mcp__tavily-mcp__tavily_crawl` to extract content from many pages on a site.

## When to use

- The user needs content from many pages on the same site.
- You need to inspect a docs section or site section in bulk.
- Map plus individual extract would be too slow or incomplete.

Use `tavily-mcp-map` first when you need to understand site structure before extracting content. Use `tavily-mcp-extract` for a small number of known URLs.

## Parameters

| Parameter | Use |
| --- | --- |
| `url` | Root URL to crawl. |
| `instructions` | Natural language guidance for semantic focus. |
| `max_depth` | Crawl depth from the root. Start with 1 unless deeper coverage is needed. |
| `max_breadth` | Maximum links to follow per level. |
| `limit` | Total pages to process. Always set a sensible cap for broad sites. |
| `select_paths` | Regex patterns for included paths. |
| `select_domains` | Regex patterns for allowed domains or subdomains. |
| `allow_external` | Whether external links may be returned. Disable for site-scoped crawls. |
| `extract_depth` | `basic` for normal pages, `advanced` for dynamic or table-heavy pages. |
| `format` | `markdown` by default, or `text` for plain text. |
| `include_favicon` | Include favicon URLs. |

## Patterns

### Crawl a docs section

Use a docs URL as the root, set `select_paths` for the docs path, and set a conservative `limit`.

### Semantic focused crawl

Use `instructions` when you only need pages or chunks relevant to a topic such as authentication, pricing, or API reference.

### Dynamic site crawl

Use `extract_depth: advanced` if pages are JavaScript-heavy or content is missing from a basic crawl.

## Tips

- Start conservative: low depth, limited breadth, and a clear page cap.
- Use path and domain filters to avoid unrelated content.
- Prefer `map` before crawl when you are unsure how large the site is.
- Prefer `extract` over crawl for one or two known URLs.
- Keep the user informed if a crawl may return a lot of content.