minhngoc25a Tavily MCP crawl
Use mcp__tavily-mcp__tavily_crawl to extract content from many pages on a site.
When to use
- The user needs content from many pages on the same site.
- You need to inspect a docs section or site section in bulk.
- Map plus individual extract would be too slow or incomplete.
Use tavily-mcp-map first when you need to understand site structure before extracting content. Use tavily-mcp-extract for a small number of known URLs.
Parameters
| Parameter | Use |
|---|---|
url |
Root URL to crawl. |
instructions |
Natural language guidance for semantic focus. |
max_depth |
Crawl depth from the root. Start with 1 unless deeper coverage is needed. |
max_breadth |
Maximum links to follow per level. |
limit |
Total pages to process. Always set a sensible cap for broad sites. |
select_paths |
Regex patterns for included paths. |
select_domains |
Regex patterns for allowed domains or subdomains. |
allow_external |
Whether external links may be returned. Disable for site-scoped crawls. |
extract_depth |
basic for normal pages, advanced for dynamic or table-heavy pages. |
format |
markdown by default, or text for plain text. |
include_favicon |
Include favicon URLs. |
Patterns
Crawl a docs section
Use a docs URL as the root, set select_paths for the docs path, and set a conservative limit.
Semantic focused crawl
Use instructions when you only need pages or chunks relevant to a topic such as authentication, pricing, or API reference.
Dynamic site crawl
Use extract_depth: advanced if pages are JavaScript-heavy or content is missing from a basic crawl.
Tips
- Start conservative: low depth, limited breadth, and a clear page cap.
- Use path and domain filters to avoid unrelated content.
- Prefer
mapbefore crawl when you are unsure how large the site is. - Prefer
extractover crawl for one or two known URLs. - Keep the user informed if a crawl may return a lot of content.