Screaming Frog

Screaming Frog SEO Spider CLI

Automate Screaming Frog SEO Spider for technical SEO audits, crawl exports, and report generation via command line.

Configuration

Set the executable path in .env:

SCREAMING_FROG_PATH=C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe

Default paths (fallback if env var not set):

• Windows: C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe
• Mac: /Applications/Screaming Frog SEO Spider.app/Contents/MacOS/ScreamingFrogSEOSpiderCli
• Linux: screamingfrogseospider

Quick Start

Basic Crawl with Exports

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Internal:All,Response Codes:All,Page Titles:All" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite

Load Existing Crawl

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --load-crawl "path/to/crawl.seospider" \
  --export-tabs "Internal:All" \
  --output-folder "output/"

Core Command Structure

Every Screaming Frog CLI command requires three components:

1. Scan Source - What to crawl or load
2. Export Type - What data to export
3. Output - Where to save results

Essential Arguments

Scan Source (choose one)

Always Include

Export Format

Export Types

1. Export Tabs (--export-tabs)

Export main interface tabs. Use comma-separated list.

Available tabs:

Internal:All, External:All, Images:All, Content:All
Page Titles:All, Meta Description:All, Response Codes:All
Canonicals:All, H1:All, H2:All, AMP:All, Analytics:All
Directives:All, Hreflang:All, JavaScript:All, CSS:All
Links:All Inlinks, Links:All Outlinks, Security:All

Example:

--export-tabs "Internal:All,Page Titles:All,Meta Description:All,H1:All,Response Codes:All,Canonicals:All"

2. Save Reports (--save-report)

Generate specific analysis reports.

Available reports:

Crawl Overview, Issues Overview
Canonicals:Canonical Chains, Canonicals:Non-Indexable Canonicals
Redirects:All Redirects, Redirects:Redirect Chains
SERP Summary, Orphan Pages
PageSpeed:Opportunities, PageSpeed:Diagnostics
Structured Data:Validation Errors

Example:

--save-report "Crawl Overview,Redirects:All Redirects,Issues Overview"

3. Bulk Export (--bulk-export)

Export detailed data sets.

Available exports:

Links:All Inlinks, Links:All Outlinks
Response Codes:All, Response Codes:Client Errors (4xx)
Response Codes:Server Errors (5xx), Response Codes:Redirects (3xx)
Content:All, Content:Duplicate
Images:All, Images:Missing Alt Text
Canonicals:All, Canonicals:Non-Indexable Canonicals
Structured Data:All, Structured Data:Validation Errors
Issues:All

Example:

--bulk-export "Links:All Inlinks,Response Codes:Client Errors (4xx),Issues:All"

API Integrations

Enable external data integrations with flags:

Note: API credentials must be configured in Screaming Frog GUI first.

Special Features

Common Use Cases

1. Full Technical Audit

Complete site crawl with all technical exports:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Internal:All,Response Codes:All,Page Titles:All,Meta Description:All,H1:All,Canonicals:All,Directives:All" \
  --save-report "Crawl Overview,Issues Overview,Redirects:All Redirects,Canonicals:Canonical Chains" \
  --bulk-export "Issues:All" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite

2. Content Audit

Focus on on-page SEO elements:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Page Titles:All,Meta Description:All,H1:All,H2:All,Content:All" \
  --save-report "Crawl Overview" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite

3. Link Analysis

Internal and external link audit:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Internal:All,External:All,Links:All Inlinks,Links:All Outlinks" \
  --bulk-export "Links:All Inlinks,Links:All Outlinks" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite

4. Response Code & Redirect Audit

Focus on HTTP status codes and redirects:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Response Codes:All" \
  --save-report "Redirects:All Redirects,Redirects:Redirect Chains" \
  --bulk-export "Response Codes:Client Errors (4xx),Response Codes:Server Errors (5xx)" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite

5. Performance Audit (with PageSpeed/Lighthouse)

Core Web Vitals and performance metrics:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --use-pagespeed \
  --lighthouse \
  --save-report "PageSpeed:Opportunities,PageSpeed:Diagnostics,Crawl Overview" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite

6. Sitemap Generation

Create XML sitemaps from crawl:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --create-sitemap \
  --create-images-sitemap \
  --output-folder "output/" \
  --overwrite

7. Compare Crawls (Change Detection)

Compare current crawl with baseline:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --compare "path/to/baseline.seospider" \
  --export-tabs "Internal:All" \
  --save-report "Crawl Overview" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite

8. Structured Data Validation

Validate schema markup:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --structured-data \
  --export-tabs "Structured Data:All" \
  --bulk-export "Structured Data:Validation Errors" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite

9. Re-export from Saved Crawl

Load existing crawl and export different data:

"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --load-crawl "output/site.seospider" \
  --export-tabs "Internal:All,Page Titles:All" \
  --bulk-export "Links:All Inlinks" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite

Output Files

Screaming Frog creates multiple files based on export type:

Help Commands

List available options:

# All options
ScreamingFrogSEOSpiderCli.exe --help

# Available export tabs
ScreamingFrogSEOSpiderCli.exe --help export-tabs

# Available reports
ScreamingFrogSEOSpiderCli.exe --help save-report

# Available bulk exports
ScreamingFrogSEOSpiderCli.exe --help bulk-export

Tips

1. Always use --headless for automation - prevents GUI from launching
2. Use --overwrite to avoid file conflict errors
3. Prefer xlsx format for Excel compatibility and multiple sheets
4. Configure APIs in GUI first before using integration flags
5. Large sites: Screaming Frog has memory limits - consider URL limits in config
6. Save crawl files for re-analysis without re-crawling
7. Check license: Some features require paid license

Processing Outputs

After export, use transform or xlsx skills to process the data:

import pandas as pd

# Load exported Excel file
df = pd.read_excel("output/internal_all.xlsx")

# Filter for issues
errors_df = df[df['Status Code'] >= 400]

Argument Reference

*One of --crawl or --load-crawl is required.