# Crawling

> Multi-page crawling and site mapping. This skill should be used when the user asks to crawl an entire website, map all URLs on a site, discover site structure, or scrape multiple pages from a domain.

- Skill: `agents-store/crawling` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add agents-store/crawling`
- Raw SKILL.md: https://api.skillmd.com/api/skills/agents-store/crawling/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: agents-store (https://skillmd.com/u/agents-store)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/agents-store/crawling

---


# Site Crawling

This skill covers crawling entire websites and mapping site structures — async crawling, URL patterns, depth control, and progress monitoring.

## Available Tools

| Tool | Description |
|------|-------------|
| `crawl_site` | Start an async crawl of a website |
| `get_crawl_status` | Check crawl progress and get results |
| `map_site` | Discover all URLs on a site |

## Map Site (URL Discovery)

Quickly discover all URLs on a site without downloading content.

```
Tool: map_site
Input: {
  "url": "https://example.com"
}

Returns: Array of all discovered URLs on the site.
```

Use this BEFORE crawling to understand site structure and set appropriate include/exclude patterns.

## Start a Crawl

```
Tool: crawl_site
Input: {
  "url": "https://example.com",
  "max_pages": 50,
  "include_patterns": ["/blog/*", "/docs/*"],
  "exclude_patterns": ["/admin/*", "/api/*"],
  "formats": ["markdown"]
}

Returns: { "crawl_id": "crawl_xxx", "status": "crawling" }
```

### Configuration Options

| Parameter | Type | Description |
|-----------|------|-------------|
| url | string | Starting URL |
| max_pages | number | Maximum pages to crawl (default: 50) |
| max_depth | number | Maximum link depth from start URL |
| include_patterns | string[] | URL patterns to include (glob) |
| exclude_patterns | string[] | URL patterns to exclude (glob) |
| formats | string[] | Output formats per page |

## Check Crawl Status

Crawls are async — poll for progress.

```
Tool: get_crawl_status
Input: {
  "crawl_id": "crawl_xxx"
}

Returns: {
  "status": "crawling" | "completed" | "failed",
  "pages_crawled": 35,
  "total_pages": 50,
  "data": [...] // page content when completed
}
```

## Common Workflows

### Crawl Blog Section
```
1. map_site(url) -> Discover all URLs
2. Identify blog URL pattern (e.g., /blog/*)
3. crawl_site(url, include_patterns=["/blog/*"], max_pages=100)
4. get_crawl_status(crawl_id) -> Poll until complete
5. Process all blog post content
```

### Full Site Crawl with Monitoring
```
1. map_site(url) -> Estimate site size
2. crawl_site(url, max_pages=200) -> Start crawl
3. get_crawl_status(crawl_id) -> Check progress periodically
4. When complete: process results
```

### Selective Documentation Crawl
```
1. map_site(url) -> See URL structure
2. crawl_site(url, include_patterns=["/docs/*"], exclude_patterns=["/docs/api/*"])
3. get_crawl_status(crawl_id) -> Wait for completion
4. Extract and organize documentation content
```

## URL Pattern Syntax

Patterns use glob-style matching:
- `*` matches any characters within a path segment
- `**` matches any characters across path segments
- `/blog/*` matches `/blog/post-1`, `/blog/post-2`
- `/docs/**` matches `/docs/guide/intro`, `/docs/api/v2/ref`

## Additional crawl_site Parameters

| Parameter | Type | Description |
|-----------|------|-------------|
| `limit` | number | Alias for max_pages |
| `maxDepth` | number | Maximum crawl depth |
| `maxDiscoveryDepth` | number | Maximum depth for URL discovery |
| `allowExternalLinks` | boolean | Follow links to other domains |
| `deduplicateSimilarURLs` | boolean | Skip pages with similar content |

## Error Handling and Recovery

### Handle Failed Crawls
```
1. crawl_site(url, max_pages=100) → Start crawl
2. get_crawl_status(crawl_id) → Check status
3. If status == "failed":
   - Verify URL is accessible: scrape_url(url) first
   - Reduce max_pages and retry
   - Try with more specific include_patterns
```

### Large Site Strategy
```
For sites with 1000+ pages:
1. map_site(url) → Get full URL list (fast, no content)
2. Identify sections: /blog/*, /docs/*, /products/*
3. Crawl each section separately:
   crawl_site(url, include_patterns=["/blog/*"], max_pages=200)
   crawl_site(url, include_patterns=["/docs/*"], max_pages=200)
4. Process results per section
```

### Incremental Crawling
```
For monitoring site changes over time:
1. crawl_site(url, max_pages=50) → Initial crawl
2. Store results with URLs and timestamps
3. Later: crawl_site(url, max_pages=50) → New crawl
4. Compare results to identify new/changed pages
```

## Best Practices

1. **Map before crawl** — use `map_site` to understand structure first
2. **Set max_pages** — always limit crawls to avoid scraping too much
3. **Use include patterns** — focus on relevant sections
4. **Exclude admin/API paths** — avoid non-content URLs
5. **Monitor progress** — poll `get_crawl_status` periodically
6. **Start small** — test with max_pages=10, then increase
7. **Use markdown format** — most efficient for content processing
8. **Split large sites** — crawl by section rather than all at once
9. **Handle failures** — check status and retry with adjusted parameters
10. **Deduplicate** — use `deduplicateSimilarURLs` to skip similar pages