# Screaming Frog

> Screaming Frog SEO Spider CLI for site crawls, technical audits, and exports. Use to crawl sites, export tabs (internal URLs, response codes, canonicals, titles), generate SEO reports (overview, redirects, orphan pages), bulk export links/issues, load .seospider files, generate XML sitemaps, compare crawls, integrate PageSpeed/Lighthouse, or validate structured data. Outputs to ./output/.

- Skill: `buzzmatic/screaming-frog` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add buzzmatic/screaming-frog`
- Raw SKILL.md: https://api.skillmd.com/api/skills/buzzmatic/screaming-frog/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- Author: Buzzmatic (https://skillmd.com/u/buzzmatic)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/buzzmatic/screaming-frog

---


# Screaming Frog SEO Spider CLI

Automate Screaming Frog SEO Spider for technical SEO audits, crawl exports, and report generation via command line.

## Configuration

Set the executable path in `.env`:
```
SCREAMING_FROG_PATH=C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe
```

**Default paths (fallback if env var not set):**
- Windows: `C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe`
- Mac: `/Applications/Screaming Frog SEO Spider.app/Contents/MacOS/ScreamingFrogSEOSpiderCli`
- Linux: `screamingfrogseospider`

## Quick Start

### Basic Crawl with Exports

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Internal:All,Response Codes:All,Page Titles:All" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite
```

### Load Existing Crawl

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --load-crawl "path/to/crawl.seospider" \
  --export-tabs "Internal:All" \
  --output-folder "output/"
```

## Core Command Structure

Every Screaming Frog CLI command requires three components:

1. **Scan Source** - What to crawl or load
2. **Export Type** - What data to export
3. **Output** - Where to save results

## Essential Arguments

### Scan Source (choose one)

| Argument | Description |
|----------|-------------|
| `--crawl [URL]` | Crawl a website (include https://) |
| `--load-crawl [file.seospider]` | Load previously saved crawl file |

### Always Include

| Argument | Description |
|----------|-------------|
| `--headless` | Run without GUI (required for automation) |
| `--output-folder [PATH]` | Where to save exports |
| `--overwrite` | Overwrite existing files |

### Export Format

| Argument | Description |
|----------|-------------|
| `--export-format csv` | CSV format (default) |
| `--export-format xlsx` | Excel format (recommended) |
| `--export-format xls` | Legacy Excel format |
| `--export-format gsheet` | Google Sheets (requires auth) |

## Export Types

### 1. Export Tabs (`--export-tabs`)

Export main interface tabs. Use comma-separated list.

**Available tabs:**
```
Internal:All, External:All, Images:All, Content:All
Page Titles:All, Meta Description:All, Response Codes:All
Canonicals:All, H1:All, H2:All, AMP:All, Analytics:All
Directives:All, Hreflang:All, JavaScript:All, CSS:All
Links:All Inlinks, Links:All Outlinks, Security:All
```

**Example:**
```bash
--export-tabs "Internal:All,Page Titles:All,Meta Description:All,H1:All,Response Codes:All,Canonicals:All"
```

### 2. Save Reports (`--save-report`)

Generate specific analysis reports.

**Available reports:**
```
Crawl Overview, Issues Overview
Canonicals:Canonical Chains, Canonicals:Non-Indexable Canonicals
Redirects:All Redirects, Redirects:Redirect Chains
SERP Summary, Orphan Pages
PageSpeed:Opportunities, PageSpeed:Diagnostics
Structured Data:Validation Errors
```

**Example:**
```bash
--save-report "Crawl Overview,Redirects:All Redirects,Issues Overview"
```

### 3. Bulk Export (`--bulk-export`)

Export detailed data sets.

**Available exports:**
```
Links:All Inlinks, Links:All Outlinks
Response Codes:All, Response Codes:Client Errors (4xx)
Response Codes:Server Errors (5xx), Response Codes:Redirects (3xx)
Content:All, Content:Duplicate
Images:All, Images:Missing Alt Text
Canonicals:All, Canonicals:Non-Indexable Canonicals
Structured Data:All, Structured Data:Validation Errors
Issues:All
```

**Example:**
```bash
--bulk-export "Links:All Inlinks,Response Codes:Client Errors (4xx),Issues:All"
```

## API Integrations

Enable external data integrations with flags:

| Flag | Integration |
|------|-------------|
| `--use-google-search-console` | GSC data overlay |
| `--use-pagespeed` | PageSpeed Insights |
| `--use-google-analytics` | Universal Analytics |
| `--use-google-analytics-4` | GA4 |
| `--use-ahrefs` | Ahrefs metrics |
| `--use-majestic` | Majestic backlink data |
| `--use-mozscape` | Moz DA/PA |

**Note:** API credentials must be configured in Screaming Frog GUI first.

## Special Features

| Argument | Description |
|----------|-------------|
| `--lighthouse` | Run Lighthouse/Core Web Vitals checks |
| `--structured-data` | Validate schema markup |
| `--compare [file.seospider]` | Compare with previous crawl |
| `--create-sitemap` | Generate XML sitemap |
| `--create-images-sitemap` | Generate image sitemap |
| `--config [file]` | Use custom config file |
| `--timestamped-output` | Create timestamped output folders |

## Common Use Cases

### 1. Full Technical Audit

Complete site crawl with all technical exports:

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Internal:All,Response Codes:All,Page Titles:All,Meta Description:All,H1:All,Canonicals:All,Directives:All" \
  --save-report "Crawl Overview,Issues Overview,Redirects:All Redirects,Canonicals:Canonical Chains" \
  --bulk-export "Issues:All" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite
```

### 2. Content Audit

Focus on on-page SEO elements:

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Page Titles:All,Meta Description:All,H1:All,H2:All,Content:All" \
  --save-report "Crawl Overview" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite
```

### 3. Link Analysis

Internal and external link audit:

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Internal:All,External:All,Links:All Inlinks,Links:All Outlinks" \
  --bulk-export "Links:All Inlinks,Links:All Outlinks" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite
```

### 4. Response Code & Redirect Audit

Focus on HTTP status codes and redirects:

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --export-tabs "Response Codes:All" \
  --save-report "Redirects:All Redirects,Redirects:Redirect Chains" \
  --bulk-export "Response Codes:Client Errors (4xx),Response Codes:Server Errors (5xx)" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite
```

### 5. Performance Audit (with PageSpeed/Lighthouse)

Core Web Vitals and performance metrics:

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --use-pagespeed \
  --lighthouse \
  --save-report "PageSpeed:Opportunities,PageSpeed:Diagnostics,Crawl Overview" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite
```

### 6. Sitemap Generation

Create XML sitemaps from crawl:

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --create-sitemap \
  --create-images-sitemap \
  --output-folder "output/" \
  --overwrite
```

### 7. Compare Crawls (Change Detection)

Compare current crawl with baseline:

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --compare "path/to/baseline.seospider" \
  --export-tabs "Internal:All" \
  --save-report "Crawl Overview" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite
```

### 8. Structured Data Validation

Validate schema markup:

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --crawl https://example.com \
  --structured-data \
  --export-tabs "Structured Data:All" \
  --bulk-export "Structured Data:Validation Errors" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite
```

### 9. Re-export from Saved Crawl

Load existing crawl and export different data:

```bash
"C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe" \
  --headless \
  --load-crawl "output/site.seospider" \
  --export-tabs "Internal:All,Page Titles:All" \
  --bulk-export "Links:All Inlinks" \
  --output-folder "output/" \
  --export-format xlsx \
  --overwrite
```

## Output Files

Screaming Frog creates multiple files based on export type:

| Export Type | Output Files |
|-------------|--------------|
| `--export-tabs` | `{tab_name}.xlsx` (e.g., `internal_all.xlsx`) |
| `--save-report` | `{report_name}.xlsx` (e.g., `crawl_overview.xlsx`) |
| `--bulk-export` | `{export_name}.xlsx` (e.g., `all_inlinks.xlsx`) |
| `--create-sitemap` | `sitemap.xml` |

## Help Commands

List available options:

```bash
# All options
ScreamingFrogSEOSpiderCli.exe --help

# Available export tabs
ScreamingFrogSEOSpiderCli.exe --help export-tabs

# Available reports
ScreamingFrogSEOSpiderCli.exe --help save-report

# Available bulk exports
ScreamingFrogSEOSpiderCli.exe --help bulk-export
```

## Tips

1. **Always use `--headless`** for automation - prevents GUI from launching
2. **Use `--overwrite`** to avoid file conflict errors
3. **Prefer xlsx format** for Excel compatibility and multiple sheets
4. **Configure APIs in GUI first** before using integration flags
5. **Large sites**: Screaming Frog has memory limits - consider URL limits in config
6. **Save crawl files** for re-analysis without re-crawling
7. **Check license**: Some features require paid license

## Processing Outputs

After export, use transform or xlsx skills to process the data:

```python
import pandas as pd

# Load exported Excel file
df = pd.read_excel("output/internal_all.xlsx")

# Filter for issues
errors_df = df[df['Status Code'] >= 400]
```

## Argument Reference

| Argument | Type | Required | Description |
|----------|------|----------|-------------|
| `--headless` | flag | Yes | Run without GUI |
| `--crawl` | URL | * | Website URL to crawl |
| `--load-crawl` | path | * | Load .seospider file |
| `--export-tabs` | list | No | Tabs to export |
| `--save-report` | list | No | Reports to generate |
| `--bulk-export` | list | No | Bulk data exports |
| `--output-folder` | path | Yes | Output directory |
| `--export-format` | string | No | csv/xlsx/xls/gsheet |
| `--overwrite` | flag | No | Overwrite existing |
| `--timestamped-output` | flag | No | Timestamped folders |
| `--config` | path | No | Custom config file |
| `--use-google-search-console` | flag | No | Enable GSC |
| `--use-pagespeed` | flag | No | Enable PageSpeed |
| `--use-google-analytics-4` | flag | No | Enable GA4 |
| `--use-ahrefs` | flag | No | Enable Ahrefs |
| `--lighthouse` | flag | No | Enable Lighthouse |
| `--structured-data` | flag | No | Validate schema |
| `--compare` | path | No | Compare with crawl |
| `--create-sitemap` | flag | No | Generate XML sitemap |
| `--create-images-sitemap` | flag | No | Generate image sitemap |

*One of `--crawl` or `--load-crawl` is required.