Technical SEO Audit
When to Use
- Use when the user wants a technical SEO review focused on crawlability, indexability, performance, or rendering.
- Use when auditing robots.txt, canonicalization, JavaScript SEO, Core Web Vitals, or AI crawler access.
- Use when the task is infrastructure- and implementation-oriented rather than content-focused.
Categories
1. Crawlability
- robots.txt: exists, valid, not blocking important resources
- XML sitemap: exists, referenced in robots.txt, valid format
- Noindex tags: intentional vs accidental
- Crawl depth: important pages within 3 clicks of homepage
- JavaScript rendering: check if critical content requires JS execution
- Crawl budget: for large sites (>10k pages), efficiency matters
AI Crawler Management
As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.
Known AI crawlers:
| Crawler |
Company |
robots.txt token |
Purpose |
| GPTBot |
OpenAI |
GPTBot |
Model training |
| ChatGPT-User |
OpenAI |
ChatGPT-User |
Real-time browsing |
| ClaudeBot |
Anthropic |
ClaudeBot |
Model training |
| PerplexityBot |
Perplexity |
PerplexityBot |
Search index + training |
| Bytespider |
ByteDance |
Bytespider |
Model training |
| Google-Extended |
Google |
Google-Extended |
Gemini training (NOT search) |
| CCBot |
Common Crawl |
CCBot |
Open dataset |
Key distinctions:
- Blocking
Google-Extended prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use Googlebot)
- Blocking
GPTBot prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (ChatGPT-User)
- ~3-5% of websites now use AI-specific robots.txt rules
Example, selective AI crawler blocking:
# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
# Allow all other crawlers (including Googlebot for search)
User-agent: *
Allow: /
Recommendation: Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the seo-geo skill for full AI visibility optimization.
2. Indexability
- Canonical tags: self-referencing, no conflicts with noindex
- Duplicate content: near-duplicates, parameter URLs, www vs non-www
- Thin content: pages below minimum word counts per type
- Pagination: rel=next/prev or load-more pattern
- Hreflang: correct for multi-language/multi-region sites
- Index bloat: unnecessary pages consuming crawl budget
3. Security
- HTTPS: enforced, valid SSL certificate, no mixed content
- Security headers:
- Content-Security-Policy (CSP)
- Strict-Transport-Security (HSTS)
- X-Frame-Options
- X-Content-Type-Options
- Referrer-Policy
- HSTS preload: check preload list inclusion for high-security sites
4. URL Structure
- Clean URLs: descriptive, hyphenated, no query parameters for content
- Hierarchy: logical folder structure reflecting site architecture
- Redirects: no chains (max 1 hop), 301 for permanent moves
- URL length: flag >100 characters
- Trailing slashes: consistent usage
5. Mobile Optimization
- Responsive design: viewport meta tag, responsive CSS
- Touch targets: minimum 48x48px with 8px spacing
- Font size: minimum 16px base
- No horizontal scroll
- Mobile-first indexing: Google indexes mobile version. Mobile-first indexing is 100% complete as of July 5, 2024. Google now crawls and indexes ALL websites exclusively with the mobile Googlebot user-agent.
6. Core Web Vitals
- LCP (Largest Contentful Paint): target <2.5s
- INP (Interaction to Next Paint): target <200ms
- INP replaced FID on March 12, 2024. FID was fully removed from all Chrome tools (CrUX API, PageSpeed Insights, Lighthouse) on September 9, 2024. Do NOT reference FID anywhere.
- CLS (Cumulative Layout Shift): target <0.1
- Evaluation uses 75th percentile of real user data
- Use PageSpeed Insights API or CrUX data if MCP available
7. Structured Data
- Detection: JSON-LD (preferred), Microdata, RDFa
- Validation against Google's supported types
- See seo-schema skill for full analysis
8. JavaScript Rendering
- Check if content visible in initial HTML vs requires JS
- Identify client-side rendered (CSR) vs server-side rendered (SSR)
- Flag SPA frameworks (React, Vue, Angular) that may cause indexing issues
- Verify dynamic rendering setup if applicable
JavaScript SEO: Canonical & Indexing Guidance (December 2025)
Google updated its JavaScript SEO documentation in December 2025 with critical clarifications:
- Canonical conflicts: If a canonical tag in raw HTML differs from one injected by JavaScript, Google may use EITHER one. Ensure canonical tags are identical between server-rendered HTML and JS-rendered output.
- noindex with JavaScript: If raw HTML contains
<meta name="robots" content="noindex"> but JavaScript removes it, Google MAY still honor the noindex from raw HTML. Serve correct robots directives in the initial HTML response.
- Non-200 status codes: Google does NOT render JavaScript on pages returning non-200 HTTP status codes. Any content or meta tags injected via JS on error pages will be invisible to Googlebot.
- Structured data in JavaScript: Product, Article, and other structured data injected via JS may face delayed processing. For time-sensitive structured data (especially e-commerce Product markup), include it in the initial server-rendered HTML.
Best practice: Serve critical SEO elements (canonical, meta robots, structured data, title, meta description) in the initial server-rendered HTML rather than relying on JavaScript injection.
9. IndexNow Protocol
- Check if site supports IndexNow for Bing, Yandex, Naver
- Supported by search engines other than Google
- Recommend implementation for faster indexing on non-Google engines
Output
Technical Score: XX/100
Category Breakdown
| Category |
Status |
Score |
| Crawlability |
pass/warn/fail |
XX/100 |
| Indexability |
pass/warn/fail |
XX/100 |
| Security |
pass/warn/fail |
XX/100 |
| URL Structure |
pass/warn/fail |
XX/100 |
| Mobile |
pass/warn/fail |
XX/100 |
| Core Web Vitals |
pass/warn/fail |
XX/100 |
| Structured Data |
pass/warn/fail |
XX/100 |
| JS Rendering |
pass/warn/fail |
XX/100 |
| IndexNow |
pass/warn/fail |
XX/100 |
Critical Issues (fix immediately)
High Priority (fix within 1 week)
Medium Priority (fix within 1 month)
Low Priority (backlog)
DataForSEO Integration (Optional)
If DataForSEO MCP tools are available, use on_page_instant_pages for real page analysis (status codes, page timing, broken links, on-page checks), on_page_lighthouse for Lighthouse audits (performance, accessibility, SEO scores), and domain_analytics_technologies_domain_technologies for technology stack detection.
Error Handling
| Scenario |
Action |
| URL unreachable |
Report connection error with status code. Suggest verifying URL, checking DNS resolution, and confirming the site is publicly accessible. |
| robots.txt not found |
Note that no robots.txt was detected at the root domain. Recommend creating one with appropriate directives. Continue audit on remaining categories. |
| HTTPS not configured |
Flag as a critical issue. Report whether HTTP is served without redirect, mixed content exists, or SSL certificate is missing/expired. |
| Core Web Vitals data unavailable |
Note that CrUX data is not available (common for low-traffic sites). Suggest using Lighthouse lab data as a proxy and recommend increasing traffic before re-testing. |
Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
Source: sickn33/agentic-awesome-skills → skills/seo-technical/SKILL.md
Also appears in: sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills/skills/seo-technical/SKILL.md, sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills-claude/skills/seo-technical/SKILL.md
1---
2name: seo-technical
3description: "Audit technical SEO across crawlability, indexability, security, URLs, mobile, Core Web Vitals, structured data, JavaScript rendering, and related platform signals like robots.txt and AI crawler access."
4allowed-tools: "Read Grep Glob Bash WebFetch"
5category: marketing-and-growth
6source_repo: sickn33/agentic-awesome-skills
7source_path: "skills/seo-technical/SKILL.md"
8source_url: https://github.com/sickn33/agentic-awesome-skills/blob/HEAD/skills/seo-technical/SKILL.md
9metadata:
10 source: https://skillmd.com/skills/sickn33/seo-technical
11---
12
13
14# Technical SEO Audit
15
16## When to Use
17- Use when the user wants a technical SEO review focused on crawlability, indexability, performance, or rendering.
18- Use when auditing robots.txt, canonicalization, JavaScript SEO, Core Web Vitals, or AI crawler access.
19- Use when the task is infrastructure- and implementation-oriented rather than content-focused.
20
21## Categories
22
23### 1. Crawlability
24- robots.txt: exists, valid, not blocking important resources
25- XML sitemap: exists, referenced in robots.txt, valid format
26- Noindex tags: intentional vs accidental
27- Crawl depth: important pages within 3 clicks of homepage
28- JavaScript rendering: check if critical content requires JS execution
29- Crawl budget: for large sites (>10k pages), efficiency matters
30
31#### AI Crawler Management
32
33As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.
34
35**Known AI crawlers:**
36
37| Crawler | Company | robots.txt token | Purpose |
38|---------|---------|-----------------|---------|
39| GPTBot | OpenAI | `GPTBot` | Model training |
40| ChatGPT-User | OpenAI | `ChatGPT-User` | Real-time browsing |
41| ClaudeBot | Anthropic | `ClaudeBot` | Model training |
42| PerplexityBot | Perplexity | `PerplexityBot` | Search index + training |
43| Bytespider | ByteDance | `Bytespider` | Model training |
44| Google-Extended | Google | `Google-Extended` | Gemini training (NOT search) |
45| CCBot | Common Crawl | `CCBot` | Open dataset |
46
47**Key distinctions:**
48- Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
49- Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`)
50- ~3-5% of websites now use AI-specific robots.txt rules
51
52**Example, selective AI crawler blocking:**
53```
54# Allow search indexing, block AI training crawlers
55User-agent: GPTBot
56Disallow: /
57
58User-agent: Google-Extended
59Disallow: /
60
61User-agent: Bytespider
62Disallow: /
63
64# Allow all other crawlers (including Googlebot for search)
65User-agent: *
66Allow: /
67```
68
69**Recommendation:** Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the `seo-geo` skill for full AI visibility optimization.
70
71### 2. Indexability
72- Canonical tags: self-referencing, no conflicts with noindex
73- Duplicate content: near-duplicates, parameter URLs, www vs non-www
74- Thin content: pages below minimum word counts per type
75- Pagination: rel=next/prev or load-more pattern
76- Hreflang: correct for multi-language/multi-region sites
77- Index bloat: unnecessary pages consuming crawl budget
78
79### 3. Security
80- HTTPS: enforced, valid SSL certificate, no mixed content
81- Security headers:
82 - Content-Security-Policy (CSP)
83 - Strict-Transport-Security (HSTS)
84 - X-Frame-Options
85 - X-Content-Type-Options
86 - Referrer-Policy
87- HSTS preload: check preload list inclusion for high-security sites
88
89### 4. URL Structure
90- Clean URLs: descriptive, hyphenated, no query parameters for content
91- Hierarchy: logical folder structure reflecting site architecture
92- Redirects: no chains (max 1 hop), 301 for permanent moves
93- URL length: flag >100 characters
94- Trailing slashes: consistent usage
95
96### 5. Mobile Optimization
97- Responsive design: viewport meta tag, responsive CSS
98- Touch targets: minimum 48x48px with 8px spacing
99- Font size: minimum 16px base
100- No horizontal scroll
101- Mobile-first indexing: Google indexes mobile version. **Mobile-first indexing is 100% complete as of July 5, 2024.** Google now crawls and indexes ALL websites exclusively with the mobile Googlebot user-agent.
102
103### 6. Core Web Vitals
104- **LCP** (Largest Contentful Paint): target <2.5s
105- **INP** (Interaction to Next Paint): target <200ms
106 - INP replaced FID on March 12, 2024. FID was fully removed from all Chrome tools (CrUX API, PageSpeed Insights, Lighthouse) on September 9, 2024. Do NOT reference FID anywhere.
107- **CLS** (Cumulative Layout Shift): target <0.1
108- Evaluation uses 75th percentile of real user data
109- Use PageSpeed Insights API or CrUX data if MCP available
110
111### 7. Structured Data
112- Detection: JSON-LD (preferred), Microdata, RDFa
113- Validation against Google's supported types
114- See seo-schema skill for full analysis
115
116### 8. JavaScript Rendering
117- Check if content visible in initial HTML vs requires JS
118- Identify client-side rendered (CSR) vs server-side rendered (SSR)
119- Flag SPA frameworks (React, Vue, Angular) that may cause indexing issues
120- Verify dynamic rendering setup if applicable
121
122#### JavaScript SEO: Canonical & Indexing Guidance (December 2025)
123
124Google updated its JavaScript SEO documentation in December 2025 with critical clarifications:
125
1261. **Canonical conflicts:** If a canonical tag in raw HTML differs from one injected by JavaScript, Google may use EITHER one. Ensure canonical tags are identical between server-rendered HTML and JS-rendered output.
1272. **noindex with JavaScript:** If raw HTML contains `<meta name="robots" content="noindex">` but JavaScript removes it, Google MAY still honor the noindex from raw HTML. Serve correct robots directives in the initial HTML response.
1283. **Non-200 status codes:** Google does NOT render JavaScript on pages returning non-200 HTTP status codes. Any content or meta tags injected via JS on error pages will be invisible to Googlebot.
1294. **Structured data in JavaScript:** Product, Article, and other structured data injected via JS may face delayed processing. For time-sensitive structured data (especially e-commerce Product markup), include it in the initial server-rendered HTML.
130
131**Best practice:** Serve critical SEO elements (canonical, meta robots, structured data, title, meta description) in the initial server-rendered HTML rather than relying on JavaScript injection.
132
133### 9. IndexNow Protocol
134- Check if site supports IndexNow for Bing, Yandex, Naver
135- Supported by search engines other than Google
136- Recommend implementation for faster indexing on non-Google engines
137
138## Output
139
140### Technical Score: XX/100
141
142### Category Breakdown
143| Category | Status | Score |
144|----------|--------|-------|
145| Crawlability | pass/warn/fail | XX/100 |
146| Indexability | pass/warn/fail | XX/100 |
147| Security | pass/warn/fail | XX/100 |
148| URL Structure | pass/warn/fail | XX/100 |
149| Mobile | pass/warn/fail | XX/100 |
150| Core Web Vitals | pass/warn/fail | XX/100 |
151| Structured Data | pass/warn/fail | XX/100 |
152| JS Rendering | pass/warn/fail | XX/100 |
153| IndexNow | pass/warn/fail | XX/100 |
154
155### Critical Issues (fix immediately)
156### High Priority (fix within 1 week)
157### Medium Priority (fix within 1 month)
158### Low Priority (backlog)
159
160## DataForSEO Integration (Optional)
161
162If DataForSEO MCP tools are available, use `on_page_instant_pages` for real page analysis (status codes, page timing, broken links, on-page checks), `on_page_lighthouse` for Lighthouse audits (performance, accessibility, SEO scores), and `domain_analytics_technologies_domain_technologies` for technology stack detection.
163
164## Error Handling
165
166| Scenario | Action |
167|----------|--------|
168| URL unreachable | Report connection error with status code. Suggest verifying URL, checking DNS resolution, and confirming the site is publicly accessible. |
169| robots.txt not found | Note that no robots.txt was detected at the root domain. Recommend creating one with appropriate directives. Continue audit on remaining categories. |
170| HTTPS not configured | Flag as a critical issue. Report whether HTTP is served without redirect, mixed content exists, or SSL certificate is missing/expired. |
171| Core Web Vitals data unavailable | Note that CrUX data is not available (common for low-traffic sites). Suggest using Lighthouse lab data as a proxy and recommend increasing traffic before re-testing. |
172
173## Limitations
174- Use this skill only when the task clearly matches the scope described above.
175- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
176- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
177
178---
179
180**Source:** [`sickn33/agentic-awesome-skills`](https://github.com/sickn33/agentic-awesome-skills) → `skills/seo-technical/SKILL.md`
181
182**Also appears in:** `sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills/skills/seo-technical/SKILL.md`, `sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills-claude/skills/seo-technical/SKILL.md`