Crawlability Audit

Crawlability Audit

Install

Save this file as ~/.claude/skills/crawlability-audit/SKILL.md, or .claude/skills/crawlability-audit/SKILL.md to scope it to one repo. Claude Code auto-discovers it. Invoke with /crawlability-audit or by asking "why isn't this page getting indexed?".

Why this exists

SEO advice is mostly about content. The failures that actually cost the most are mechanical, silent, and take minutes to fix once seen: a prefix rule that blocks more than anyone intended, a sitemap advertising URLs that robots forbids, an error page returning 200 so crawlers file it as thin content instead of retrying.

None of these announce themselves. Traffic is simply flat, and everyone blames the copy.

Rank by what it costs. A rule blocking your signup or seller page outranks a missing meta description on a blog post, always.

Step 1 — Read robots.txt as a machine would

curl -s https://<host>/robots.txt

Every Disallow is a PREFIX match, not a path match. This is the single most expensive misunderstanding in the file:

• Disallow: /sell blocks /sell, and also /sellers, /selling, /seller/anyone, /sell-your-thing. Whole sections vanish.
• Disallow: /sell/ blocks only children, leaving /sell itself crawlable.

For each Disallow, enumerate the real routes it matches. Read the router or route directory; do not guess:

rg --files -g '*/page.tsx' -g '*/route.ts' -g 'pages/**' | sort

Then ask, per matched route: is this genuinely private? A rule written for an authenticated dashboard very often also covers the public marketing page that shares its prefix. That is the bug, and it is usually invisible because the public page still works perfectly for humans.

Step 2 — Cross-check robots against every sitemap

Fetch every sitemap, including nested ones:

curl -s https://<host>/sitemap.xml
curl -s https://<host>/robots.txt | grep -i sitemap

Two contradictions to hunt:

1. URLs in a sitemap that robots disallows. You are asking a crawler to index pages you have forbidden it to read. Search consoles report this as a warning most people never open.
2. Sitemaps that exist but are never declared. Generated and unlinked means undiscoverable. Compare generated sitemap routes against the Sitemap: lines.

comm -12 <(sitemap paths, sorted) <(disallowed prefixes expanded, sorted)

Any overlap is a finding.

Step 3 — Check what status codes crawlers actually receive

A page that renders an error but returns 200 is worse than one returning 503. On 200 the crawler stores an empty page as your content; on 5xx it retains what it knows and retries.

curl -s -o /dev/null -w "%{http_code}\n" https://<host>/<path>
curl -s https://<host>/<path> | head -c 2000

Compare status to CONTENT, not to whether the request succeeded. Flag:

• 200 responses whose body is an error state, a spinner, or empty.
• 200 responses containing a client-side render abort with no server content.
• Sitemaps returning 200 while listing zero real URLs. This actively tells a crawler your catalog was withdrawn, every time it is fetched.
• Soft 404s: "not found" text served with a 200.

Step 4 — Metadata on the pages that earn money

For each high-intent route (catalog, pricing, signup, top products):

curl -s https://<host>/<path> | grep -oE '<title>[^<]*</title>|<meta name="description"[^>]*>|<link rel="canonical"[^>]*>'

Flag, in priority order:

1. Duplicate title or description across routes. Two pages competing for one query with identical words. Extremely common on catalog pages that inherit a site-wide default and get missed because they "look fine".
2. Missing canonical on anything reachable by more than one URL. Query parameters for filtering and sorting mint unbounded near-duplicates. Pin the canonical to the bare path.
3. Missing structured data on commercial pages. Product, Offer, and price markup is how both rich results and AI answer engines read a listing. Check whether a helper already exists and is simply not wired to the page it describes; that is a frequent and cheap win.

Step 5 — Internal links, since that is how discovery actually works

Sitemaps are a hint. Internal links are the signal.

curl -s https://<host>/ | grep -oE 'href="/[^"]*"' | sort -u

Flag redirect hops on primary navigation. If every catalog card points at a tracking route that 302s to the real page, no page has direct internal links. Tracking is a fair reason; just do not let the pages built to RANK inherit it.

Step 6 — Report

Ranked by revenue impact, not by count:

BLOCKED FROM CRAWLERS
  <path>  <- matched by "Disallow: <rule>", is actually <public purpose>

CONTRADICTIONS
  <n> URLs are in a sitemap AND disallowed
  <n> sitemaps generated but never declared in robots.txt

WRONG STATUS CODES
  <path>  200 but body is <error/empty>

METADATA
  <path>  duplicate title with <path>
  <path>  no canonical, reachable via <n> query permutations

Lead with the single most expensive line and say what it costs in plain words: "your seller signup page has been invisible to Google" beats "robots.txt misconfiguration".

Rules

• Verify against the LIVE site. The deployed robots.txt is frequently older than the source file, and the live one is the only one crawlers obey.
• Never recommend blocking a page to "save crawl budget" unless it is genuinely private or genuinely duplicate. Crawl budget is not a real constraint for most sites and this advice deletes traffic.
• Never recommend adding structured data describing things that are not on the page. Invented ratings and fake availability get manual actions.
• Prefer one fixed prefix rule over twenty new pages of content.

From Toolbay. Free to use, modify, and share. Keep this line and others can find it too.