Advertools Spider

Advertools Spider

Comprehensive web crawling powered by advertools.spider (Scrapy-based). Automatically extracts SEO elements and supports custom data extraction with maximum flexibility.

Quick Start

Discovery Crawl (Follow Links)

Crawl entire site starting from seed URL(s):

python scripts/crawl.py \
  --url https://example.com \
  --follow-links

List Crawl (Specific URLs)

Crawl only specified URLs without following links:

python scripts/crawl.py \
  --url-list urls.txt

URLs file format (one per line):

https://example.com/page1
https://example.com/page2
https://example.com/page3

Crawl Modes

Discovery Mode (--follow-links): Start with seed URL(s) and recursively discover pages by following links. Use for site-wide audits, comprehensive crawls, and exploration.

List Mode (default): Crawl only the specified URLs without discovery. Use for targeted analysis, competitor research, SERP URL audits, or when you have a predetermined list.

Extracted Data

Default SEO Elements

Automatically extracted without configuration:

• Metadata: title, meta description, meta keywords, canonical, hreflang
• Headings: h1, h2, h3, h4, h5, h6 (with counts)
• Open Graph: og:title, og:description, og:image, og:type, etc.
• Twitter Cards: twitter:card, twitter:title, twitter:description, etc.
• Structured Data: JSON-LD extraction
• Links: all internal/external links with anchor text, rel attributes
• Images: src, alt text, dimensions
• Technical: status codes, response time, redirect chains, IP addresses, response headers
• Content: body text, word count

Custom Extraction

Extract specific data using CSS or XPath selectors:

CSS Selectors:

python scripts/crawl.py \
  --url https://shop.com \
  --follow-links \
  --css-selectors '{"price": ".product-price", "rating": ".star-rating", "reviews": ".review-count"}'

XPath Selectors:

python scripts/crawl.py \
  --url https://example.com \
  --xpath-selectors '{"author": "//article//span[@class=\"author\"]/text()", "date": "//time/@datetime"}'

Combine both for complex extraction:

python scripts/crawl.py \
  --url https://site.com \
  --css-selectors '{"title": "h1.product-title"}' \
  --xpath-selectors '{"sku": "//meta[@itemprop=\"sku\"]/@content"}'

URL Filtering

Control which URLs get crawled using regex or parameter patterns:

Include/Exclude by Regex

Include only product and category pages:

--include-url-regex "/(products|category)/"

Exclude admin, login, and cart pages:

--exclude-url-regex "/(admin|login|cart|checkout)/"

Include/Exclude by URL Parameters

Exclude URLs with tracking parameters:

--exclude-url-params "utm_source,utm_medium,sessionid,ref"

Only crawl URLs with specific parameters:

--include-url-params "category,product_id"

Domain Control

Restrict crawling to specific domains (useful for cross-domain crawls):

python scripts/crawl.py \
  --url https://example.com \
  --follow-links \
  --allowed-domains "example.com,subdomain.example.com"

Advanced Configuration

Custom Scrapy Settings

Control concurrency, depth, delays, and other Scrapy settings:

python scripts/crawl.py \
  --url https://example.com \
  --follow-links \
  --custom-settings '{
    "CONCURRENT_REQUESTS": 16,
    "DEPTH_LIMIT": 3,
    "DOWNLOAD_DELAY": 0.5,
    "ROBOTSTXT_OBEY": true,
    "USER_AGENT": "CustomBot/1.0"
  }'

Common settings:

• CONCURRENT_REQUESTS: Number of parallel requests (default: 16)
• DEPTH_LIMIT: Maximum crawl depth from seed URLs
• DOWNLOAD_DELAY: Delay between requests in seconds
• ROBOTSTXT_OBEY: Respect robots.txt (default: true)
• CLOSESPIDER_PAGECOUNT: Stop after N pages
• USER_AGENT: Custom user agent string

Column Control

Filter output columns using regex patterns:

Keep only specific columns:

--keep-columns "^(url|title|meta_desc|h1|status)$"

Discard noisy columns:

--discard-columns "^(jsonld_|og_|twitter_)"

Custom Output Name

Specify output filename (default: crawl_TIMESTAMP):

--output-name "competitor_analysis"
# Output: output/competitor_analysis.jl

Common Use Cases

Technical SEO Audit

python scripts/crawl.py \
  --url https://example.com \
  --follow-links \
  --custom-settings '{"DEPTH_LIMIT": 5, "CLOSESPIDER_PAGECOUNT": 10000}'

Competitor Content Analysis

python scripts/crawl.py \
  --url-list competitor_urls.txt \
  --css-selectors '{"word_count": "article", "author": ".author-name", "publish_date": "time.published"}'

E-commerce Product Scraping

python scripts/crawl.py \
  --url https://shop.com/products \
  --follow-links \
  --include-url-regex "/product/" \
  --css-selectors '{"price": ".price", "availability": ".stock-status", "sku": ".product-code"}' \
  --custom-settings '{"CONCURRENT_REQUESTS": 8, "DOWNLOAD_DELAY": 1}'

SERP URL Audit (from GSC data)

# Assume GSC data exported to urls.txt
python scripts/crawl.py \
  --url-list urls.txt
  # Crawls each URL, extracts default SEO elements

Sitemap-based Crawl

# Extract URLs from sitemap first, then crawl
python scripts/crawl.py \
  --url-list sitemap_urls.txt \
  --custom-settings '{"CONCURRENT_REQUESTS": 32}'

Output Format

Data saved as JSONL (JSON Lines) to output/{output_name}.jl

Each line is a JSON object representing one crawled page. Load with pandas:

import pandas as pd
df = pd.read_json('output.jl', lines=True)

Script Output

The script returns JSON summary:

{
  "success": true,
  "output_file": "output/crawl_20250102_143022.jl",
  "stats": {
    "total_pages": 1247,
    "total_size_mb": 15.3,
    "columns": ["url", "title", "meta_desc", "h1", "status", ...],
    "status_codes": {"200": 1198, "404": 23, "301": 26},
    "crawl_mode": "discovery",
    "seed_urls": 1
  }
}

Parameter Reference

Tips

1. Start small: Test with --custom-settings '{"CLOSESPIDER_PAGECOUNT": 100}' before full crawls
2. Respect rate limits: Use DOWNLOAD_DELAY and CONCURRENT_REQUESTS appropriately
3. Monitor progress: Check file size during long crawls to estimate completion
4. Combine filters: Use both regex and parameter filtering for precise control
5. Custom extraction: Test selectors on sample pages before full crawl
6. Large crawls: Discovery mode can generate thousands of pages - set limits