Technical SEO Audit
Find the technical reasons a site cannot be crawled, indexed, or understood, and fix them in priority order.
A technical audit does not make content better or guarantee rankings. It removes obstacles. Report findings, not predictions about traffic.
When to Use
- Pre-launch check for a new site or redesign
- After a migration (domain, CMS, framework, URL structure)
- Pages missing from the index or showing as "Discovered/Crawled - currently not indexed" in Search Console
- Duplicate URLs, wrong pages ranking, wrong language version shown
- Field Core Web Vitals failing in Search Console or CrUX
When NOT to Use
- The question is about content quality or topic strategy (see
programmatic-seo-guardrails)
- The question is about AI assistants and answer engines (see
ai-search-visibility)
Workflow
Step 1: Gather inputs
Ask for, or collect:
| Input |
Why |
Domain(s) and preferred host (https://www. vs apex) |
Canonical host decisions |
| Search Console access (Pages/Indexing, Sitemaps, Core Web Vitals reports) |
Ground truth for what Google sees |
| List of important URLs (templates: home, category, product, article) |
Audit by template, not page by page |
| Framework and rendering mode (SSR, SSG, CSR) |
JavaScript rendering risk |
| Recent changes (migration date, deploys) |
Correlate with drops |
Audit one representative URL per template. Most technical issues are template-wide.
Step 2: Crawl access (robots.txt)
curl -sI https://example.com/robots.txt # expect 200
curl -s https://example.com/robots.txt
Check against Google's documented behavior:
- Google supports
user-agent, allow, disallow, and sitemap. crawl-delay is not supported by Google.
- Path values are case-sensitive; field names are not.
- A crawler obeys only one group: the most specific matching
user-agent group. Rules in a * group do not also apply to Googlebot if a Googlebot group exists.
- The most specific (longest) matching rule wins; on a tie, the least restrictive rule wins.
- Max file size Google processes is 500 KiB.
- A 4xx (except 429) on robots.txt is treated as "no restrictions". A 5xx initially stops crawling.
Minimal, valid example:
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
Sitemap: https://www.example.com/sitemap.xml
Red flags: Disallow: / left over from staging; blocking CSS/JS needed for rendering; blocking URLs you want de-indexed (see Step 3).
Step 3: Indexability
For each template URL:
curl -sI https://www.example.com/products/widget | grep -iE "^(HTTP|x-robots-tag|location|link)"
curl -s https://www.example.com/products/widget | grep -iE '<meta[^>]+name="(robots|googlebot)"|rel="canonical"'
| Check |
Expected |
| Status code |
200 for live pages, 404/410 for gone, 301/308 for moved |
noindex (meta or X-Robots-Tag) |
Only on pages you intend to exclude |
| robots.txt + noindex conflict |
A page blocked by robots.txt cannot have its noindex seen; it may still be indexed as a bare URL. Unblock it so noindex works |
| Soft 404s |
"Not found" pages must not return 200 |
Use Search Console URL Inspection for the rendered HTML Google actually saw.
Step 4: Canonicalization and redirects
Google's signal strength: redirects and rel="canonical" are strong signals; sitemap inclusion is a weak one. rel="canonical" is a hint, not a directive.
- Use absolute canonical URLs:
<link rel="canonical" href="https://www.example.com/dresses/green/">
- Do not send mixed signals (sitemap lists URL A, canonical says URL B).
- Do not use robots.txt or
noindex to choose a canonical within a site.
- Permanent moves: 301 or 308. Google treats 302, 303, and 307 as temporary and does not use them as a canonical signal.
- Collapse redirect chains to a single hop where you control the server.
curl -sIL http://example.com/old-page | grep -iE "^(HTTP|location)"
Step 5: XML sitemaps
- Each sitemap file: max 50,000 URLs or 50 MB uncompressed; use a sitemap index beyond that.
- Absolute URLs only.
- Google ignores
<priority> and <changefreq>. Keep <lastmod> accurate (significant content changes only) or omit it.
- Include only canonical, indexable, 200-status URLs.
- Submit in Search Console and reference via a
Sitemap: line in robots.txt.
Step 6: International (hreflang)
- Pick one method: HTML
<link rel="alternate" hreflang>, HTTP Link header, or sitemap <xhtml:link>.
- Every version lists itself and every other version; if two pages do not reference each other, Google ignores the annotation.
- Codes: ISO 639-1 language, optional ISO 3166-1 Alpha-2 region (
en-GB). A region alone is invalid.
- Use
x-default for the fallback or language selector page.
- Fully qualified URLs.
<link rel="alternate" hreflang="en-us" href="https://www.example.com/us/" />
<link rel="alternate" hreflang="en-gb" href="https://www.example.com/uk/" />
<link rel="alternate" hreflang="x-default" href="https://www.example.com/" />
Step 7: JavaScript rendering
Google crawls, then renders, then indexes. Check that:
- Links are
<a href="..."> elements; client routing uses the History API, not #fragments.
- Missing content returns a real 404 (or a JS-injected
noindex), not a 200 "not found" view.
noindex is not present in the initial HTML if you expect JS to remove it; Google may skip rendering.
- Critical content and metadata appear in server HTML where possible. Server-side or pre-rendering also helps crawlers that do not run JavaScript.
Step 8: Core Web Vitals
Assess field data at the 75th percentile, mobile and desktop separately.
| Metric |
Good |
Poor |
| LCP (Largest Contentful Paint) |
<= 2.5 s |
> 4.0 s |
| INP (Interaction to Next Paint) |
<= 200 ms |
> 500 ms |
| CLS (Cumulative Layout Shift) |
<= 0.1 |
> 0.25 |
INP replaced FID as a Core Web Vital in March 2024. Flag any tooling or report still optimizing FID. Use Search Console's Core Web Vitals report or CrUX for field data. Lighthouse is lab data: useful for debugging, but it cannot tell you what real users experienced.
Step 9: Structured data
- Prefer JSON-LD. Mark up only content visible on the page.
- Validate with the Rich Results Test and URL Inspection.
- Correct markup does not guarantee a rich result.
- FAQ rich results are limited to well-known authoritative government and health sites, and HowTo rich results are deprecated (Google, August and September 2023). Do not promise these to other sites. Unused markup does not cause problems.
Step 10: Report
| # | Severity | Template | Finding | Evidence | Fix | Doc |
|---|----------|----------|---------|----------|-----|-----|
| 1 | Critical | /products/* | X-Robots-Tag: noindex in prod | curl -I output | Remove header in CDN rule | block-indexing |
Severity: Critical (blocks indexing site-wide or for a key template), High (wrong canonical, broken hreflang, poor CWV on key templates), Medium/Low (hygiene).
Best Practices
- Audit by template, then confirm with URL Inspection; do not trust one crawler tool alone.
- Quote the exact header, tag, or line as evidence for every finding.
- Fix blockers (robots, noindex, status codes) before optimizations (CWV, structured data).
- Never "fix" duplicates with robots.txt; use redirects or canonicals.
- Re-check after deploy; CDN and edge rules often add or strip headers.
- Do not promise ranking or traffic outcomes. Report what changed and what to monitor.
References
Checked 2026-11-04:
1---2name: technical-seo-audit3description: |4 Run a structured technical SEO audit of a website: crawl access (robots.txt), indexability (noindex, status codes), canonicalization, redirects, XML sitemaps, hreflang, JavaScript rendering, Core Web Vitals, and structured data validation. Produces a prioritized findings list tied to Google Search Central documentation.5 Use this skill when a site is not being indexed, traffic dropped after a migration, a new site is about to launch, or someone asks for an SEO health check.6 Activate when: technical SEO, SEO audit, site audit, robots.txt, sitemap, canonical, noindex, hreflang, redirects, Core Web Vitals, LCP, INP, CLS, indexing problems, crawl errors.7metadata:8 source: https://skillmd.com/skills/latestaiagents/technical-seo-audit9---1011# Technical SEO Audit1213**Find the technical reasons a site cannot be crawled, indexed, or understood, and fix them in priority order.**1415A technical audit does not make content better or guarantee rankings. It removes obstacles. Report findings, not predictions about traffic.1617## When to Use1819- Pre-launch check for a new site or redesign20- After a migration (domain, CMS, framework, URL structure)21- Pages missing from the index or showing as "Discovered/Crawled - currently not indexed" in Search Console22- Duplicate URLs, wrong pages ranking, wrong language version shown23- Field Core Web Vitals failing in Search Console or CrUX2425## When NOT to Use2627- The question is about content quality or topic strategy (see `programmatic-seo-guardrails`)28- The question is about AI assistants and answer engines (see `ai-search-visibility`)2930## Workflow3132### Step 1: Gather inputs3334Ask for, or collect:3536| Input | Why |37|-------|-----|38| Domain(s) and preferred host (`https://www.` vs apex) | Canonical host decisions |39| Search Console access (Pages/Indexing, Sitemaps, Core Web Vitals reports) | Ground truth for what Google sees |40| List of important URLs (templates: home, category, product, article) | Audit by template, not page by page |41| Framework and rendering mode (SSR, SSG, CSR) | JavaScript rendering risk |42| Recent changes (migration date, deploys) | Correlate with drops |4344Audit one representative URL per template. Most technical issues are template-wide.4546### Step 2: Crawl access (robots.txt)4748```bash49curl -sI https://example.com/robots.txt # expect 20050curl -s https://example.com/robots.txt51```5253Check against Google's documented behavior:5455- Google supports `user-agent`, `allow`, `disallow`, and `sitemap`. `crawl-delay` is not supported by Google.56- Path values are case-sensitive; field names are not.57- A crawler obeys only one group: the most specific matching `user-agent` group. Rules in a `*` group do not also apply to `Googlebot` if a `Googlebot` group exists.58- The most specific (longest) matching rule wins; on a tie, the least restrictive rule wins.59- Max file size Google processes is 500 KiB.60- A 4xx (except 429) on robots.txt is treated as "no restrictions". A 5xx initially stops crawling.6162Minimal, valid example:6364```text65User-agent: *66Disallow: /cart/67Disallow: /search68Allow: /search/help6970Sitemap: https://www.example.com/sitemap.xml71```7273Red flags: `Disallow: /` left over from staging; blocking CSS/JS needed for rendering; blocking URLs you want de-indexed (see Step 3).7475### Step 3: Indexability7677For each template URL:7879```bash80curl -sI https://www.example.com/products/widget | grep -iE "^(HTTP|x-robots-tag|location|link)"81curl -s https://www.example.com/products/widget | grep -iE '<meta[^>]+name="(robots|googlebot)"|rel="canonical"'82```8384| Check | Expected |85|-------|----------|86| Status code | 200 for live pages, 404/410 for gone, 301/308 for moved |87| `noindex` (meta or `X-Robots-Tag`) | Only on pages you intend to exclude |88| robots.txt + noindex conflict | A page blocked by robots.txt cannot have its `noindex` seen; it may still be indexed as a bare URL. Unblock it so `noindex` works |89| Soft 404s | "Not found" pages must not return 200 |9091Use Search Console URL Inspection for the rendered HTML Google actually saw.9293### Step 4: Canonicalization and redirects9495Google's signal strength: redirects and `rel="canonical"` are strong signals; sitemap inclusion is a weak one. `rel="canonical"` is a hint, not a directive.9697- Use absolute canonical URLs: `<link rel="canonical" href="https://www.example.com/dresses/green/">`98- Do not send mixed signals (sitemap lists URL A, canonical says URL B).99- Do not use robots.txt or `noindex` to choose a canonical within a site.100- Permanent moves: 301 or 308. Google treats 302, 303, and 307 as temporary and does not use them as a canonical signal.101- Collapse redirect chains to a single hop where you control the server.102103```bash104curl -sIL http://example.com/old-page | grep -iE "^(HTTP|location)"105```106107### Step 5: XML sitemaps108109- Each sitemap file: max 50,000 URLs or 50 MB uncompressed; use a sitemap index beyond that.110- Absolute URLs only.111- Google ignores `<priority>` and `<changefreq>`. Keep `<lastmod>` accurate (significant content changes only) or omit it.112- Include only canonical, indexable, 200-status URLs.113- Submit in Search Console and reference via a `Sitemap:` line in robots.txt.114115### Step 6: International (hreflang)116117- Pick one method: HTML `<link rel="alternate" hreflang>`, HTTP `Link` header, or sitemap `<xhtml:link>`.118- Every version lists itself and every other version; if two pages do not reference each other, Google ignores the annotation.119- Codes: ISO 639-1 language, optional ISO 3166-1 Alpha-2 region (`en-GB`). A region alone is invalid.120- Use `x-default` for the fallback or language selector page.121- Fully qualified URLs.122123```html124<link rel="alternate" hreflang="en-us" href="https://www.example.com/us/" />125<link rel="alternate" hreflang="en-gb" href="https://www.example.com/uk/" />126<link rel="alternate" hreflang="x-default" href="https://www.example.com/" />127```128129### Step 7: JavaScript rendering130131Google crawls, then renders, then indexes. Check that:132133- Links are `<a href="...">` elements; client routing uses the History API, not `#fragments`.134- Missing content returns a real 404 (or a JS-injected `noindex`), not a 200 "not found" view.135- `noindex` is not present in the initial HTML if you expect JS to remove it; Google may skip rendering.136- Critical content and metadata appear in server HTML where possible. Server-side or pre-rendering also helps crawlers that do not run JavaScript.137138### Step 8: Core Web Vitals139140Assess field data at the 75th percentile, mobile and desktop separately.141142| Metric | Good | Poor |143|--------|------|------|144| LCP (Largest Contentful Paint) | <= 2.5 s | > 4.0 s |145| INP (Interaction to Next Paint) | <= 200 ms | > 500 ms |146| CLS (Cumulative Layout Shift) | <= 0.1 | > 0.25 |147148INP replaced FID as a Core Web Vital in March 2024. Flag any tooling or report still optimizing FID. Use Search Console's Core Web Vitals report or CrUX for field data. Lighthouse is lab data: useful for debugging, but it cannot tell you what real users experienced.149150### Step 9: Structured data151152- Prefer JSON-LD. Mark up only content visible on the page.153- Validate with the Rich Results Test and URL Inspection.154- Correct markup does not guarantee a rich result.155- FAQ rich results are limited to well-known authoritative government and health sites, and HowTo rich results are deprecated (Google, August and September 2023). Do not promise these to other sites. Unused markup does not cause problems.156157### Step 10: Report158159```markdown160| # | Severity | Template | Finding | Evidence | Fix | Doc |161|---|----------|----------|---------|----------|-----|-----|162| 1 | Critical | /products/* | X-Robots-Tag: noindex in prod | curl -I output | Remove header in CDN rule | block-indexing |163```164165Severity: **Critical** (blocks indexing site-wide or for a key template), **High** (wrong canonical, broken hreflang, poor CWV on key templates), **Medium/Low** (hygiene).166167## Best Practices1681691. Audit by template, then confirm with URL Inspection; do not trust one crawler tool alone.1702. Quote the exact header, tag, or line as evidence for every finding.1713. Fix blockers (robots, noindex, status codes) before optimizations (CWV, structured data).1724. Never "fix" duplicates with robots.txt; use redirects or canonicals.1735. Re-check after deploy; CDN and edge rules often add or strip headers.1746. Do not promise ranking or traffic outcomes. Report what changed and what to monitor.175176## References177178Checked 2026-11-04:179180- Robots.txt interpretation: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt181- Block indexing with noindex: https://developers.google.com/search/docs/crawling-indexing/block-indexing182- Canonicalization: https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls183- Redirects: https://developers.google.com/search/docs/crawling-indexing/301-redirects184- Sitemaps: https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap185- Localized versions (hreflang): https://developers.google.com/search/docs/specialty/international/localized-versions186- JavaScript SEO basics: https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics187- Core Web Vitals: https://web.dev/articles/vitals, https://web.dev/articles/lcp, https://web.dev/articles/inp, https://web.dev/articles/cls188- INP becomes a Core Web Vital: https://web.dev/blog/inp-cwv-launch189- Structured data guidelines: https://developers.google.com/search/docs/appearance/structured-data/sd-policies190- FAQ and HowTo changes: https://developers.google.com/search/blog/2023/08/howto-faq-changes191- Robots Exclusion Protocol: RFC 9309, https://www.rfc-editor.org/rfc/rfc9309