# Advertools Spider

> Crawl/scrape websites using advertools.spider for SEO analysis, content extraction, and technical audits. Outputs JSONL to ./output/. Use to crawl sites, extract page elements (titles, meta, headings, structured data, custom CSS/XPath selectors), perform tech audits, scrape competitors, or crawl URL lists from sitemaps/CSVs/GSC. Discovery mode (follow links) or list mode.

- Skill: `buzzmatic/advertools-spider` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add buzzmatic/advertools-spider`
- Raw SKILL.md: https://api.skillmd.com/api/skills/buzzmatic/advertools-spider/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Product & Planning
- Author: Buzzmatic (https://skillmd.com/u/buzzmatic)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/buzzmatic/advertools-spider

---


# Advertools Spider

Comprehensive web crawling powered by advertools.spider (Scrapy-based). Automatically extracts SEO elements and supports custom data extraction with maximum flexibility.

## Quick Start

### Discovery Crawl (Follow Links)

Crawl entire site starting from seed URL(s):

```bash
python scripts/crawl.py \
  --url https://example.com \
  --follow-links
```

### List Crawl (Specific URLs)

Crawl only specified URLs without following links:

```bash
python scripts/crawl.py \
  --url-list urls.txt
```

URLs file format (one per line):
```
https://example.com/page1
https://example.com/page2
https://example.com/page3
```

## Crawl Modes

**Discovery Mode** (`--follow-links`): Start with seed URL(s) and recursively discover pages by following links. Use for site-wide audits, comprehensive crawls, and exploration.

**List Mode** (default): Crawl only the specified URLs without discovery. Use for targeted analysis, competitor research, SERP URL audits, or when you have a predetermined list.

## Extracted Data

### Default SEO Elements

Automatically extracted without configuration:

- **Metadata**: title, meta description, meta keywords, canonical, hreflang
- **Headings**: h1, h2, h3, h4, h5, h6 (with counts)
- **Open Graph**: og:title, og:description, og:image, og:type, etc.
- **Twitter Cards**: twitter:card, twitter:title, twitter:description, etc.
- **Structured Data**: JSON-LD extraction
- **Links**: all internal/external links with anchor text, rel attributes
- **Images**: src, alt text, dimensions
- **Technical**: status codes, response time, redirect chains, IP addresses, response headers
- **Content**: body text, word count

### Custom Extraction

Extract specific data using CSS or XPath selectors:

**CSS Selectors:**
```bash
python scripts/crawl.py \
  --url https://shop.com \
  --follow-links \
  --css-selectors '{"price": ".product-price", "rating": ".star-rating", "reviews": ".review-count"}'
```

**XPath Selectors:**
```bash
python scripts/crawl.py \
  --url https://example.com \
  --xpath-selectors '{"author": "//article//span[@class=\"author\"]/text()", "date": "//time/@datetime"}'
```

Combine both for complex extraction:
```bash
python scripts/crawl.py \
  --url https://site.com \
  --css-selectors '{"title": "h1.product-title"}' \
  --xpath-selectors '{"sku": "//meta[@itemprop=\"sku\"]/@content"}'
```

## URL Filtering

Control which URLs get crawled using regex or parameter patterns:

### Include/Exclude by Regex

**Include only product and category pages:**
```bash
--include-url-regex "/(products|category)/"
```

**Exclude admin, login, and cart pages:**
```bash
--exclude-url-regex "/(admin|login|cart|checkout)/"
```

### Include/Exclude by URL Parameters

**Exclude URLs with tracking parameters:**
```bash
--exclude-url-params "utm_source,utm_medium,sessionid,ref"
```

**Only crawl URLs with specific parameters:**
```bash
--include-url-params "category,product_id"
```

## Domain Control

Restrict crawling to specific domains (useful for cross-domain crawls):

```bash
python scripts/crawl.py \
  --url https://example.com \
  --follow-links \
  --allowed-domains "example.com,subdomain.example.com"
```

## Advanced Configuration

### Custom Scrapy Settings

Control concurrency, depth, delays, and other Scrapy settings:

```bash
python scripts/crawl.py \
  --url https://example.com \
  --follow-links \
  --custom-settings '{
    "CONCURRENT_REQUESTS": 16,
    "DEPTH_LIMIT": 3,
    "DOWNLOAD_DELAY": 0.5,
    "ROBOTSTXT_OBEY": true,
    "USER_AGENT": "CustomBot/1.0"
  }'
```

Common settings:
- `CONCURRENT_REQUESTS`: Number of parallel requests (default: 16)
- `DEPTH_LIMIT`: Maximum crawl depth from seed URLs
- `DOWNLOAD_DELAY`: Delay between requests in seconds
- `ROBOTSTXT_OBEY`: Respect robots.txt (default: true)
- `CLOSESPIDER_PAGECOUNT`: Stop after N pages
- `USER_AGENT`: Custom user agent string

### Column Control

Filter output columns using regex patterns:

**Keep only specific columns:**
```bash
--keep-columns "^(url|title|meta_desc|h1|status)$"
```

**Discard noisy columns:**
```bash
--discard-columns "^(jsonld_|og_|twitter_)"
```

### Custom Output Name

Specify output filename (default: `crawl_TIMESTAMP`):

```bash
--output-name "competitor_analysis"
# Output: output/competitor_analysis.jl
```

## Common Use Cases

### Technical SEO Audit

```bash
python scripts/crawl.py \
  --url https://example.com \
  --follow-links \
  --custom-settings '{"DEPTH_LIMIT": 5, "CLOSESPIDER_PAGECOUNT": 10000}'
```

### Competitor Content Analysis

```bash
python scripts/crawl.py \
  --url-list competitor_urls.txt \
  --css-selectors '{"word_count": "article", "author": ".author-name", "publish_date": "time.published"}'
```

### E-commerce Product Scraping

```bash
python scripts/crawl.py \
  --url https://shop.com/products \
  --follow-links \
  --include-url-regex "/product/" \
  --css-selectors '{"price": ".price", "availability": ".stock-status", "sku": ".product-code"}' \
  --custom-settings '{"CONCURRENT_REQUESTS": 8, "DOWNLOAD_DELAY": 1}'
```

### SERP URL Audit (from GSC data)

```bash
# Assume GSC data exported to urls.txt
python scripts/crawl.py \
  --url-list urls.txt
  # Crawls each URL, extracts default SEO elements
```

### Sitemap-based Crawl

```bash
# Extract URLs from sitemap first, then crawl
python scripts/crawl.py \
  --url-list sitemap_urls.txt \
  --custom-settings '{"CONCURRENT_REQUESTS": 32}'
```

## Output Format

Data saved as **JSONL** (JSON Lines) to `output/{output_name}.jl`

Each line is a JSON object representing one crawled page. Load with pandas:

```python
import pandas as pd
df = pd.read_json('output.jl', lines=True)
```

## Script Output

The script returns JSON summary:

```json
{
  "success": true,
  "output_file": "output/crawl_20250102_143022.jl",
  "stats": {
    "total_pages": 1247,
    "total_size_mb": 15.3,
    "columns": ["url", "title", "meta_desc", "h1", "status", ...],
    "status_codes": {"200": 1198, "404": 23, "301": 26},
    "crawl_mode": "discovery",
    "seed_urls": 1
  }
}
```

## Parameter Reference

| Parameter | Type | Description |
|-----------|------|-------------|
| `--url` | string | Single URL or comma-separated URLs |
| `--url-list` | path | File with URLs (one per line) |
| `--follow-links` | flag | Enable discovery mode |
| `--allowed-domains` | list | Restrict to domains (comma or JSON) |
| `--css-selectors` | JSON | Custom CSS selectors: `{"name": "selector"}` |
| `--xpath-selectors` | JSON | Custom XPath selectors: `{"name": "xpath"}` |
| `--exclude-url-params` | list | Exclude URLs with these params |
| `--include-url-params` | list | Only include URLs with these params |
| `--exclude-url-regex` | regex | Exclude URLs matching pattern |
| `--include-url-regex` | regex | Only include URLs matching pattern |
| `--keep-columns` | regex | Keep only columns matching pattern |
| `--discard-columns` | regex | Remove columns matching pattern |
| `--custom-settings` | JSON | Scrapy settings dict |
| `--output-name` | string | Custom output filename |

## Tips

1. **Start small**: Test with `--custom-settings '{"CLOSESPIDER_PAGECOUNT": 100}'` before full crawls
2. **Respect rate limits**: Use `DOWNLOAD_DELAY` and `CONCURRENT_REQUESTS` appropriately
3. **Monitor progress**: Check file size during long crawls to estimate completion
4. **Combine filters**: Use both regex and parameter filtering for precise control
5. **Custom extraction**: Test selectors on sample pages before full crawl
6. **Large crawls**: Discovery mode can generate thousands of pages - set limits