Screaming Frog Tool
Screaming Frog is a desktop crawler with community-built MCP servers available. Even without MCP, the agent can analyze exported crawl data (CSV/Excel). This is your primary technical SEO audit tool.
When to Use Screaming Frog vs Other Tools
| You Need |
Use Screaming Frog |
Use Instead |
| Full technical site crawl |
✅ Most detailed crawler |
Semrush site audit (lighter) |
| Location page quality audit at scale |
✅ Best for this |
— |
| Custom data extraction (NAP, schema fields) |
✅ Unique capability |
— |
| Duplicate content detection |
✅ |
— |
| Internal linking analysis |
✅ |
— |
| Redirect chain detection |
✅ |
— |
| Schema validation per page |
✅ Custom extraction |
— |
| Missing titles/metas across hundreds of pages |
✅ |
— |
| Keyword rankings |
❌ |
Local Falcon, Semrush |
| Backlink data |
❌ |
Ahrefs |
| Search traffic data |
❌ |
GSC, GA4 |
| Citation data |
❌ |
BrightLocal |
How the Agent Interacts with Screaming Frog
With MCP Connected
The agent can trigger crawls and read results directly.
Without MCP (More Common)
The user runs the crawl locally and exports data. The agent analyzes the exported CSV/Excel files.
Tell the user what to export:
- Run the crawl in Screaming Frog
- Export: Internal > All (CSV)
- If needed: Bulk Export > specific reports (response codes, directives, schema, etc.)
- Upload the CSV to the conversation
Core Workflows
Location Page Technical Audit
When: User has a multi-location site and needs to verify all location pages are technically sound.
What to check in crawl data:
| Check |
Where to Find |
What's Wrong If... |
| Title tags |
Title 1 column |
Duplicate titles across locations, missing titles, truncated titles |
| Meta descriptions |
Meta Description 1 |
Duplicate metas, missing metas, boilerplate metas |
| H1 tags |
H1-1 column |
Missing H1, duplicate H1s, H1 doesn't include service + city |
| Word count |
Word Count column |
Under 300 words = thin content (Google may not index) |
| Status codes |
Status Code column |
404 errors, 302 redirects (should be 301), 5xx errors |
| Canonical tags |
Canonical Link Element 1 |
Self-referencing canonical missing, or canonical pointing to wrong page |
| Internal links in |
Inlinks column |
0 or 1 internal links = orphan page |
| Schema present |
Custom extraction needed |
No LocalBusiness schema on location pages |
| NAP on page |
Custom extraction needed |
Missing or inconsistent NAP |
| Page speed |
PageSpeed tab (if PSI integration enabled) |
LCP > 2.5s, CLS > 0.1 |
Custom Extraction for Local SEO
What to tell the user to configure:
LocalBusiness Schema Extraction:
- Extraction: CSS Selector or XPath
- Target:
script[type="application/ld+json"]
- This extracts the full JSON-LD block — agent can then validate schema fields
NAP Extraction:
- Business name: CSS selector for the element containing business name
- Phone: CSS selector for tel: links
- Address: CSS selector for address block
Per-page schema validation the agent should do:
- Is
@type correct? (e.g., Dentist, Plumber, LocalBusiness)
- Does
name match GBP exactly?
- Does
address match GBP exactly?
- Does
telephone match GBP exactly?
- Are
openingHoursSpecification present?
- Are
geo coordinates present and correct?
- Is
areaServed present (for SABs)?
Duplicate Content Detection
When: Multi-location sites often have boilerplate location pages with only the city name changed.
What to check:
- Near-duplicate detection (Screaming Frog has this built in)
- Word count column — if all location pages are exactly the same word count, they're probably templated
- Title tag patterns — if all titles are "[Service] in [City] | [Brand]" with identical supporting content, Google may not index them all
What "unique enough" looks like:
- Unique intro paragraph mentioning specific area landmarks, neighborhoods, demographics
- Unique service details relevant to that location
- Unique testimonials or case studies from that area
- Different photos per location
- Unique FAQ based on common questions from that area's customers
Internal Linking Analysis
When: Location pages aren't getting organic traffic and you suspect they're orphaned or poorly linked.
What to check:
- Inlinks column: How many internal pages link to each location page
- Crawl depth: How many clicks from homepage to reach each location page
- Link structure: Are location pages linked from the main navigation, footer, or buried?
Healthy internal linking for location pages:
- Every location page linked from a locations index/directory page
- Location pages linked from service pages (where relevant)
- Location pages linked from each other (nearby locations)
- Crawl depth: Max 3 clicks from homepage
Redirect Audit
When: User migrated sites, changed URLs, or has old location pages that redirected.
What to check:
- 302 redirects (should usually be 301 for permanent moves)
- Redirect chains (A → B → C → D — should be A → D)
- Redirect loops (A → B → A)
- Old location URLs that 404 instead of redirecting
Interpreting Crawl Data for Local SEO
Priority Issues (Fix First)
| Issue |
Impact |
How to Find |
| Location pages returning 404 |
Pages completely invisible |
Status Code = 404, filter to location URLs |
| Location pages not in sitemap |
Google may not discover them |
Cross-reference sitemap URLs with crawled URLs |
| Location pages blocked by robots.txt |
Google can't crawl them |
Indexability column = "Blocked by Robots.txt" |
| Location pages with noindex |
Google won't index them |
Meta Robots column contains "noindex" |
Important Issues (Fix Next)
| Issue |
Impact |
How to Find |
| Duplicate titles across locations |
Google may suppress duplicates |
Title 1 column — sort and find duplicates |
| Thin content (under 300 words) |
Google may not index |
Word Count column < 300 |
| Missing schema |
Losing structured data signals |
Custom extraction shows empty |
| Orphan pages (0-1 internal links) |
Low crawl priority, low authority |
Inlinks column = 0 or 1 |
Maintenance Issues (Fix When Possible)
| Issue |
Impact |
How to Find |
| Missing meta descriptions |
Lower CTR from search results |
Meta Description 1 = empty |
| Images without alt text |
Accessibility + minor SEO signal |
Images tab, Alt Text column empty |
| Redirect chains |
Wasted crawl budget, slow page loads |
Redirect Chains report |
What to Do Next
| What You Found |
Next Action |
Skill |
| Location pages with thin content |
Rewrite with unique, substantial content per location |
local-landing-pages |
| Missing schema on location pages |
Implement LocalBusiness schema |
local-schema |
| Duplicate titles/metas |
Rewrite with unique, keyword-targeted titles per location |
local-landing-pages |
| Orphan location pages |
Fix internal linking structure |
local-landing-pages |
| Indexing issues (noindex, robots.txt, missing from sitemap) |
Fix technical issues |
local-seo-audit |
| NAP inconsistencies found via custom extraction |
Fix on-page NAP to match GBP exactly |
gbp-optimization, local-citations |
| All technical issues documented |
Package into audit report for client |
client-deliverables |
Default next step: Screaming Frog crawl data is a goldmine but it's raw. Always prioritize: indexing blockers → content issues → structural issues → optimization opportunities.
1---2name: screaming-frog-tool3description: When the user wants a technical site audit, crawl data analysis, location page quality checks, duplicate content detection, schema validation at scale, or internal linking analysis. Trigger on "Screaming Frog," "site crawl," "technical audit," "crawl data," "broken links," "duplicate content," "location page audit," or when analyzing exported crawl CSV/Excel files.4---56# Screaming Frog Tool78Screaming Frog is a desktop crawler with community-built MCP servers available. Even without MCP, the agent can analyze exported crawl data (CSV/Excel). This is your primary technical SEO audit tool.910## When to Use Screaming Frog vs Other Tools1112| You Need | Use Screaming Frog | Use Instead |13|----------|-------------------|-------------|14| Full technical site crawl | ✅ Most detailed crawler | Semrush site audit (lighter) |15| Location page quality audit at scale | ✅ Best for this | — |16| Custom data extraction (NAP, schema fields) | ✅ Unique capability | — |17| Duplicate content detection | ✅ | — |18| Internal linking analysis | ✅ | — |19| Redirect chain detection | ✅ | — |20| Schema validation per page | ✅ Custom extraction | — |21| Missing titles/metas across hundreds of pages | ✅ | — |22| Keyword rankings | ❌ | Local Falcon, Semrush |23| Backlink data | ❌ | Ahrefs |24| Search traffic data | ❌ | GSC, GA4 |25| Citation data | ❌ | BrightLocal |2627## How the Agent Interacts with Screaming Frog2829### With MCP Connected30The agent can trigger crawls and read results directly.3132### Without MCP (More Common)33The user runs the crawl locally and exports data. The agent analyzes the exported CSV/Excel files.3435**Tell the user what to export:**361. Run the crawl in Screaming Frog372. Export: Internal > All (CSV)383. If needed: Bulk Export > specific reports (response codes, directives, schema, etc.)394. Upload the CSV to the conversation4041## Core Workflows4243### Location Page Technical Audit4445**When:** User has a multi-location site and needs to verify all location pages are technically sound.4647**What to check in crawl data:**4849| Check | Where to Find | What's Wrong If... |50|-------|--------------|-------------------|51| Title tags | Title 1 column | Duplicate titles across locations, missing titles, truncated titles |52| Meta descriptions | Meta Description 1 | Duplicate metas, missing metas, boilerplate metas |53| H1 tags | H1-1 column | Missing H1, duplicate H1s, H1 doesn't include service + city |54| Word count | Word Count column | Under 300 words = thin content (Google may not index) |55| Status codes | Status Code column | 404 errors, 302 redirects (should be 301), 5xx errors |56| Canonical tags | Canonical Link Element 1 | Self-referencing canonical missing, or canonical pointing to wrong page |57| Internal links in | Inlinks column | 0 or 1 internal links = orphan page |58| Schema present | Custom extraction needed | No LocalBusiness schema on location pages |59| NAP on page | Custom extraction needed | Missing or inconsistent NAP |60| Page speed | PageSpeed tab (if PSI integration enabled) | LCP > 2.5s, CLS > 0.1 |6162### Custom Extraction for Local SEO6364**What to tell the user to configure:**6566**LocalBusiness Schema Extraction:**67- Extraction: CSS Selector or XPath68- Target: `script[type="application/ld+json"]`69- This extracts the full JSON-LD block — agent can then validate schema fields7071**NAP Extraction:**72- Business name: CSS selector for the element containing business name73- Phone: CSS selector for tel: links74- Address: CSS selector for address block7576**Per-page schema validation the agent should do:**771. Is `@type` correct? (e.g., Dentist, Plumber, LocalBusiness)782. Does `name` match GBP exactly?793. Does `address` match GBP exactly?804. Does `telephone` match GBP exactly?815. Are `openingHoursSpecification` present?826. Are `geo` coordinates present and correct?837. Is `areaServed` present (for SABs)?8485### Duplicate Content Detection8687**When:** Multi-location sites often have boilerplate location pages with only the city name changed.8889**What to check:**90- Near-duplicate detection (Screaming Frog has this built in)91- Word count column — if all location pages are exactly the same word count, they're probably templated92- Title tag patterns — if all titles are "[Service] in [City] | [Brand]" with identical supporting content, Google may not index them all9394**What "unique enough" looks like:**95- Unique intro paragraph mentioning specific area landmarks, neighborhoods, demographics96- Unique service details relevant to that location97- Unique testimonials or case studies from that area98- Different photos per location99- Unique FAQ based on common questions from that area's customers100101### Internal Linking Analysis102103**When:** Location pages aren't getting organic traffic and you suspect they're orphaned or poorly linked.104105**What to check:**106- **Inlinks column**: How many internal pages link to each location page107- **Crawl depth**: How many clicks from homepage to reach each location page108- **Link structure**: Are location pages linked from the main navigation, footer, or buried?109110**Healthy internal linking for location pages:**111- Every location page linked from a locations index/directory page112- Location pages linked from service pages (where relevant)113- Location pages linked from each other (nearby locations)114- Crawl depth: Max 3 clicks from homepage115116### Redirect Audit117118**When:** User migrated sites, changed URLs, or has old location pages that redirected.119120**What to check:**121- 302 redirects (should usually be 301 for permanent moves)122- Redirect chains (A → B → C → D — should be A → D)123- Redirect loops (A → B → A)124- Old location URLs that 404 instead of redirecting125126## Interpreting Crawl Data for Local SEO127128### Priority Issues (Fix First)129130| Issue | Impact | How to Find |131|-------|--------|------------|132| Location pages returning 404 | Pages completely invisible | Status Code = 404, filter to location URLs |133| Location pages not in sitemap | Google may not discover them | Cross-reference sitemap URLs with crawled URLs |134| Location pages blocked by robots.txt | Google can't crawl them | Indexability column = "Blocked by Robots.txt" |135| Location pages with noindex | Google won't index them | Meta Robots column contains "noindex" |136137### Important Issues (Fix Next)138139| Issue | Impact | How to Find |140|-------|--------|------------|141| Duplicate titles across locations | Google may suppress duplicates | Title 1 column — sort and find duplicates |142| Thin content (under 300 words) | Google may not index | Word Count column < 300 |143| Missing schema | Losing structured data signals | Custom extraction shows empty |144| Orphan pages (0-1 internal links) | Low crawl priority, low authority | Inlinks column = 0 or 1 |145146### Maintenance Issues (Fix When Possible)147148| Issue | Impact | How to Find |149|-------|--------|------------|150| Missing meta descriptions | Lower CTR from search results | Meta Description 1 = empty |151| Images without alt text | Accessibility + minor SEO signal | Images tab, Alt Text column empty |152| Redirect chains | Wasted crawl budget, slow page loads | Redirect Chains report |153154## What to Do Next155156| What You Found | Next Action | Skill |157|----------------|-------------|-------|158| Location pages with thin content | Rewrite with unique, substantial content per location | `local-landing-pages` |159| Missing schema on location pages | Implement LocalBusiness schema | `local-schema` |160| Duplicate titles/metas | Rewrite with unique, keyword-targeted titles per location | `local-landing-pages` |161| Orphan location pages | Fix internal linking structure | `local-landing-pages` |162| Indexing issues (noindex, robots.txt, missing from sitemap) | Fix technical issues | `local-seo-audit` |163| NAP inconsistencies found via custom extraction | Fix on-page NAP to match GBP exactly | `gbp-optimization`, `local-citations` |164| All technical issues documented | Package into audit report for client | `client-deliverables` |165166**Default next step:** Screaming Frog crawl data is a goldmine but it's raw. Always prioritize: indexing blockers → content issues → structural issues → optimization opportunities.
garrettjsmith