# Technical SEO Audit

> Run a structured technical SEO audit of a website: crawl access (robots.txt), indexability (noindex, status codes), canonicalization, redirects, XML sitemaps, hreflang, JavaScript rendering, Core Web Vitals, and structured data validation. Produces a prioritized findings list tied to Google Search Central documentation. Use this skill when a site is not being indexed, traffic dropped after a migration, a new site is about to launch, or someone asks for an SEO health check. Activate when: technical SEO, SEO audit, site audit, robots.txt, sitemap, canonical, noindex, hreflang, redirects, Core Web Vitals, LCP, INP, CLS, indexing problems, crawl errors.

- Skill: `latestaiagents/technical-seo-audit` (Agent Skill)
- Install (CLI): `npx get-skill add latestaiagents/technical-seo-audit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/latestaiagents/technical-seo-audit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- Author: latestaiagents (https://skillmd.com/u/latestaiagents)
- Updated: 2026-11-14
- Page: https://skillmd.com/skills/latestaiagents/technical-seo-audit

---


# Technical SEO Audit

**Find the technical reasons a site cannot be crawled, indexed, or understood, and fix them in priority order.**

A technical audit does not make content better or guarantee rankings. It removes obstacles. Report findings, not predictions about traffic.

## When to Use

- Pre-launch check for a new site or redesign
- After a migration (domain, CMS, framework, URL structure)
- Pages missing from the index or showing as "Discovered/Crawled - currently not indexed" in Search Console
- Duplicate URLs, wrong pages ranking, wrong language version shown
- Field Core Web Vitals failing in Search Console or CrUX

## When NOT to Use

- The question is about content quality or topic strategy (see `programmatic-seo-guardrails`)
- The question is about AI assistants and answer engines (see `ai-search-visibility`)

## Workflow

### Step 1: Gather inputs

Ask for, or collect:

| Input | Why |
|-------|-----|
| Domain(s) and preferred host (`https://www.` vs apex) | Canonical host decisions |
| Search Console access (Pages/Indexing, Sitemaps, Core Web Vitals reports) | Ground truth for what Google sees |
| List of important URLs (templates: home, category, product, article) | Audit by template, not page by page |
| Framework and rendering mode (SSR, SSG, CSR) | JavaScript rendering risk |
| Recent changes (migration date, deploys) | Correlate with drops |

Audit one representative URL per template. Most technical issues are template-wide.

### Step 2: Crawl access (robots.txt)

```bash
curl -sI https://example.com/robots.txt        # expect 200
curl -s  https://example.com/robots.txt
```

Check against Google's documented behavior:

- Google supports `user-agent`, `allow`, `disallow`, and `sitemap`. `crawl-delay` is not supported by Google.
- Path values are case-sensitive; field names are not.
- A crawler obeys only one group: the most specific matching `user-agent` group. Rules in a `*` group do not also apply to `Googlebot` if a `Googlebot` group exists.
- The most specific (longest) matching rule wins; on a tie, the least restrictive rule wins.
- Max file size Google processes is 500 KiB.
- A 4xx (except 429) on robots.txt is treated as "no restrictions". A 5xx initially stops crawling.

Minimal, valid example:

```text
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

Sitemap: https://www.example.com/sitemap.xml
```

Red flags: `Disallow: /` left over from staging; blocking CSS/JS needed for rendering; blocking URLs you want de-indexed (see Step 3).

### Step 3: Indexability

For each template URL:

```bash
curl -sI https://www.example.com/products/widget | grep -iE "^(HTTP|x-robots-tag|location|link)"
curl -s  https://www.example.com/products/widget | grep -iE '<meta[^>]+name="(robots|googlebot)"|rel="canonical"'
```

| Check | Expected |
|-------|----------|
| Status code | 200 for live pages, 404/410 for gone, 301/308 for moved |
| `noindex` (meta or `X-Robots-Tag`) | Only on pages you intend to exclude |
| robots.txt + noindex conflict | A page blocked by robots.txt cannot have its `noindex` seen; it may still be indexed as a bare URL. Unblock it so `noindex` works |
| Soft 404s | "Not found" pages must not return 200 |

Use Search Console URL Inspection for the rendered HTML Google actually saw.

### Step 4: Canonicalization and redirects

Google's signal strength: redirects and `rel="canonical"` are strong signals; sitemap inclusion is a weak one. `rel="canonical"` is a hint, not a directive.

- Use absolute canonical URLs: `<link rel="canonical" href="https://www.example.com/dresses/green/">`
- Do not send mixed signals (sitemap lists URL A, canonical says URL B).
- Do not use robots.txt or `noindex` to choose a canonical within a site.
- Permanent moves: 301 or 308. Google treats 302, 303, and 307 as temporary and does not use them as a canonical signal.
- Collapse redirect chains to a single hop where you control the server.

```bash
curl -sIL http://example.com/old-page | grep -iE "^(HTTP|location)"
```

### Step 5: XML sitemaps

- Each sitemap file: max 50,000 URLs or 50 MB uncompressed; use a sitemap index beyond that.
- Absolute URLs only.
- Google ignores `<priority>` and `<changefreq>`. Keep `<lastmod>` accurate (significant content changes only) or omit it.
- Include only canonical, indexable, 200-status URLs.
- Submit in Search Console and reference via a `Sitemap:` line in robots.txt.

### Step 6: International (hreflang)

- Pick one method: HTML `<link rel="alternate" hreflang>`, HTTP `Link` header, or sitemap `<xhtml:link>`.
- Every version lists itself and every other version; if two pages do not reference each other, Google ignores the annotation.
- Codes: ISO 639-1 language, optional ISO 3166-1 Alpha-2 region (`en-GB`). A region alone is invalid.
- Use `x-default` for the fallback or language selector page.
- Fully qualified URLs.

```html
<link rel="alternate" hreflang="en-us" href="https://www.example.com/us/" />
<link rel="alternate" hreflang="en-gb" href="https://www.example.com/uk/" />
<link rel="alternate" hreflang="x-default" href="https://www.example.com/" />
```

### Step 7: JavaScript rendering

Google crawls, then renders, then indexes. Check that:

- Links are `<a href="...">` elements; client routing uses the History API, not `#fragments`.
- Missing content returns a real 404 (or a JS-injected `noindex`), not a 200 "not found" view.
- `noindex` is not present in the initial HTML if you expect JS to remove it; Google may skip rendering.
- Critical content and metadata appear in server HTML where possible. Server-side or pre-rendering also helps crawlers that do not run JavaScript.

### Step 8: Core Web Vitals

Assess field data at the 75th percentile, mobile and desktop separately.

| Metric | Good | Poor |
|--------|------|------|
| LCP (Largest Contentful Paint) | <= 2.5 s | > 4.0 s |
| INP (Interaction to Next Paint) | <= 200 ms | > 500 ms |
| CLS (Cumulative Layout Shift) | <= 0.1 | > 0.25 |

INP replaced FID as a Core Web Vital in March 2024. Flag any tooling or report still optimizing FID. Use Search Console's Core Web Vitals report or CrUX for field data. Lighthouse is lab data: useful for debugging, but it cannot tell you what real users experienced.

### Step 9: Structured data

- Prefer JSON-LD. Mark up only content visible on the page.
- Validate with the Rich Results Test and URL Inspection.
- Correct markup does not guarantee a rich result.
- FAQ rich results are limited to well-known authoritative government and health sites, and HowTo rich results are deprecated (Google, August and September 2023). Do not promise these to other sites. Unused markup does not cause problems.

### Step 10: Report

```markdown
| # | Severity | Template | Finding | Evidence | Fix | Doc |
|---|----------|----------|---------|----------|-----|-----|
| 1 | Critical | /products/* | X-Robots-Tag: noindex in prod | curl -I output | Remove header in CDN rule | block-indexing |
```

Severity: **Critical** (blocks indexing site-wide or for a key template), **High** (wrong canonical, broken hreflang, poor CWV on key templates), **Medium/Low** (hygiene).

## Best Practices

1. Audit by template, then confirm with URL Inspection; do not trust one crawler tool alone.
2. Quote the exact header, tag, or line as evidence for every finding.
3. Fix blockers (robots, noindex, status codes) before optimizations (CWV, structured data).
4. Never "fix" duplicates with robots.txt; use redirects or canonicals.
5. Re-check after deploy; CDN and edge rules often add or strip headers.
6. Do not promise ranking or traffic outcomes. Report what changed and what to monitor.

## References

Checked 2026-11-04:

- Robots.txt interpretation: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
- Block indexing with noindex: https://developers.google.com/search/docs/crawling-indexing/block-indexing
- Canonicalization: https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls
- Redirects: https://developers.google.com/search/docs/crawling-indexing/301-redirects
- Sitemaps: https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
- Localized versions (hreflang): https://developers.google.com/search/docs/specialty/international/localized-versions
- JavaScript SEO basics: https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics
- Core Web Vitals: https://web.dev/articles/vitals, https://web.dev/articles/lcp, https://web.dev/articles/inp, https://web.dev/articles/cls
- INP becomes a Core Web Vital: https://web.dev/blog/inp-cwv-launch
- Structured data guidelines: https://developers.google.com/search/docs/appearance/structured-data/sd-policies
- FAQ and HowTo changes: https://developers.google.com/search/blog/2023/08/howto-faq-changes
- Robots Exclusion Protocol: RFC 9309, https://www.rfc-editor.org/rfc/rfc9309

