Reference Files — Load When Relevant
| Topic |
File |
Load when... |
| Keyword research stack |
references/keyword-research.md |
Finding keywords, clustering, search volume, intent |
| SERP analysis |
references/serp-analysis.md |
Analysing SERPs, AI Overviews, content gaps, briefs |
| Technical audit |
references/technical-audit.md |
Site audits, crawling, meta tags, schema, Core Web Vitals |
| Backlink intelligence |
references/backlinks.md |
Backlink discovery, toxic scoring, competitor gaps, broken links |
| Infrastructure |
references/infrastructure.md |
Building tools, proxies, caching, deployment, APIs |
| Failure registry |
references/failures.md |
Debugging, known pitfalls, API failures, library bugs |
Project Context
If project-context.md exists in the skill directory or .claude/ folder, read it
before any task. It contains brand details, website URLs, API credentials paths,
infrastructure setup, and content defaults — avoids re-explaining context each session.
Workflow Overview
1. Keyword Research → Find opportunities, cluster by intent, assess difficulty
2. SERP Analysis → Analyse top results, track AI Overviews, generate content briefs
3. Technical Audit → Crawl site, check on-page SEO, validate schema, measure Core Web Vitals
4. Content Optimisation → Fill content gaps, optimise meta/headings/schema, improve AI extractability
5. Backlink Intelligence → Discover links, score quality, find toxic links, analyse competitor gaps
6. Monitoring → Track rankings, AI Overview rates, backlink changes, technical health
Step 1: Keyword Research
Free Stack (No Paid APIs Required)
| Source |
What it gives you |
Cost |
| Google Autocomplete |
Real-time query suggestions |
Free (use Webshare proxies) |
| YouTube Autocomplete |
Video-intent keywords |
Free |
| People Also Ask |
Question-format keywords |
Free (via Serper SERP data) |
| Related Searches |
Semantic variations |
Free (via Serper SERP data) |
| GSC API |
Queries you already rank for |
Free (service account) |
| Serper Autocomplete |
Google Trends proxy |
Free (2,500/month) |
| DataForSEO |
Volume, CPC, competition |
Optional ($1 free credit) |
AI Clustering
Send keywords to Groq (llama-3.1-8b-instant) — returns semantic topic clusters
with parent topics and sub-topics. Free tier, no credit card required.
Intent Classification
Rule-based classifier — no API needed:
- Transactional: buy, price, discount, coupon, deal, order, shop
- Commercial: best, top, review, comparison, vs, alternative
- Informational: how, what, why, when, guide, tutorial, learn
- Navigational: brand name, login, dashboard, app, official
- Local: near me, in [city], directions, open now, hours
Proxy Difficulty Score
0–100 score calculated without paid APIs:
- Count strong domains in top 10 (Wikipedia, Reddit, .gov, .edu, major brands)
- Detect AI Overview presence (+15 difficulty)
- Detect featured snippet (+10 difficulty)
- Brand detection — owned domain queries get reduced difficulty
See references/keyword-research.md for deep alphabet soup mode, monthly trends,
sparkline data, and the full implementation stack.
Step 2: SERP Analysis
Critical Rule
Google SERP scraping is dead. Google returns JS-only noscript pages from any
IP, proxy, or user agent. Do not attempt to scrape google.com directly.
Use Serper.dev API instead:
- 2,500 free searches/month
POST https://google.serper.dev/search
- Returns: organic results, AI Overview, featured snippets, PAA, knowledge panel,
videos, related searches — all structured JSON
AI Overview Tracking
Daily cron job tracks AI Overview presence per keyword. Calculate:
- Overall AI Overview rate (percentage of tracked keywords with AO)
- Per-keyword AO history (appeared/disappeared over time)
- Content cited in AI Overviews (your domain vs competitors)
Content Gap Analysis
For any target keyword:
- Pull top 5 SERP results via Serper
- Fetch and parse each page (cheerio)
- Compare: headings, word count, schema types, topics covered
- Generate gap report: what competitors cover that you don't
Content Brief Generation
Auto-generate briefs from SERP data:
- Recommended word count (median of top 5)
- Must-have topics (present in 3+ of top 5)
- Gap topics (present in competitors, missing from your page)
- Schema suggestions based on SERP features
- Recommended headings structure
See references/serp-analysis.md for SERP feature history tracking,
content brief templates, and implementation details.
Step 3: Technical Audit
Crawl Strategy
Sitemap-first crawl:
- Fetch
/sitemap.xml (handle sitemap index with nested sub-sitemaps)
- Extract all URLs
- Fetch each page, parse with cheerio
- Run per-page and site-wide checks
Per-Page Checks
| Check |
Pass Criteria |
| Title |
Present, 30–60 characters |
| Meta description |
Present, 120–160 characters |
| H1 |
Exactly one per page |
| Canonical |
Present, self-referencing or valid |
| Viewport |
Present (width=device-width) |
| OG tags |
og:title, og:description, og:image present |
| Schema |
Valid JSON-LD with required fields |
| Images |
All <img> have non-empty alt attributes |
| Word count |
≥300 for content pages |
| HTML size |
Flag if >1 MB |
Site-Wide Checks
- Duplicate titles across pages
- Duplicate meta descriptions across pages
- Orphan pages (no inbound internal links)
- Deep pages (>3 clicks from homepage)
Schema Validation
JSON-LD required fields per type:
- Article: headline, datePublished, author, image
- FAQ: mainEntity with Question + acceptedAnswer
- HowTo: name, step[] with text
- Product: name, offers with price + priceCurrency
- Organization: name, url, logo
Core Web Vitals
Google PSI API (free, no key required for basic use):
- LCP (Largest Contentful Paint) — target < 2.5s
- FID/INP (Interaction to Next Paint) — target < 200ms
- CLS (Cumulative Layout Shift) — target < 0.1
- FCP, TTFB, TBT as supplementary metrics
AI Audit Fix Suggestions
Send failing checks to Groq (llama-3.1-8b-instant):
- Generate replacement titles at correct length
- Write meta descriptions from page content
- Suggest alt text from image context
- Flag thin content pages with expansion recommendations
See references/technical-audit.md for cheerio bug workarounds,
link graph construction, and full implementation details.
Step 4: Content Optimisation
On-Page SEO Checklist
AI Visibility (GEO) Optimisation
- Structure content with clear headings and bullet points
- Include direct, quotable answers to common questions
- Add FAQ schema for question-based queries
- Use statistics and citations that AI can extract
- Ensure schema markup is comprehensive and accurate
- Monitor AI Overview citations for your domain
Step 5: Backlink Intelligence
Free Discovery Stack
| Method |
What it finds |
Limitation |
| Common Crawl CDX |
Historical backlinks across 3+ indexes |
Not real-time |
| Live HTTP verification |
Current link status |
Slow for large sets |
| Paste importers |
Links from free tools (Moz, Neil Patel, etc.) |
Manual paste step |
Quality Scoring
Each backlink scored 0–100 with A–F grade:
- Link location: content (+40) > sidebar (+15) > footer (+5) > nav (+5)
- Anchor quality: keyword-rich (+20) > branded (+15) > generic (+5) > naked URL (+3)
- Link density: fewer outbound links on page = higher score
- Page freshness: recent content scores higher
- Follow status: dofollow (+15) > nofollow (+5)
Toxic Link Detection
4-tier classification based on:
- Spam TLDs (.xyz, .top, .buzz, .click, .loan, .work)
- Numeric-heavy domains
- Excessive hyphens in domain
- Spam keywords in URL path (casino, pharmacy, payday, etc.)
Competitor Gap Analysis
Common Crawl-based diff:
- Query CC for domains linking to competitor
- Query CC for domains linking to you
- Diff = opportunity domains (link to them, not you)
See references/backlinks.md for anchor diversity (Shannon entropy),
broken outbound link checking, and paste importer formats.
Step 6: Monitoring
Recommended Cron Schedule
| Task |
Frequency |
Tool |
| Rank tracking (via Serper) |
Daily |
PM2 cron |
| AI Overview monitoring |
Daily |
PM2 cron |
| GSC data pull |
Weekly |
PM2 cron |
| Technical audit |
Weekly |
PM2 cron |
| Backlink discovery |
Monthly |
PM2 cron |
| Core Web Vitals |
Monthly |
PSI API |
Alert Triggers
- Rank drop >5 positions for tracked keyword
- AI Overview appears/disappears for tracked keyword
- New backlink from high-authority domain
- Technical issue detected (broken page, missing schema)
- Core Web Vitals regression
Quick Decision Trees
"I need more organic traffic"
Is your site technically sound?
├── No → Run Technical Audit (Step 3)
└── Yes
├── Do you have keyword targets?
│ ├── No → Run Keyword Research (Step 1)
│ └── Yes
│ ├── Are you ranking for them?
│ │ ├── No → Run SERP Analysis + Content Optimisation (Steps 2+4)
│ │ └── Yes, but low positions
│ │ ├── Content gap? → Fill gaps (Step 2)
│ │ └── Authority gap? → Backlink Intelligence (Step 5)
│ └── Are you visible in AI Overviews?
│ ├── No → GEO Optimisation (Step 4)
│ └── Yes → Monitor and maintain (Step 6)
"My rankings dropped"
When did it drop?
├── After a Google update → Check Search Status Dashboard, wait 2 weeks
├── After site changes → Technical Audit (Step 3), check canonical/redirect issues
├── Gradual decline
│ ├── Content freshness issue → Update content, add new sections
│ ├── Competitors improved → SERP Analysis (Step 2), Content Gap
│ └── Lost backlinks → Backlink monitoring, re-acquisition outreach
"I want to track AI visibility"
1. Set up daily AI Overview tracking via Serper (Step 2)
2. Monitor citation rate for your domain in AI Overviews
3. Optimise content for extractability (Step 4 — GEO)
4. Add comprehensive schema markup (Step 3)
5. Track changes weekly in monitoring dashboard (Step 6)
1---2name: seo-geo-skill3description: The complete SEO and GEO (Generative Engine Optimisation) skill — keyword research, SERP analysis, technical audits, content optimisation, backlink intelligence, and AI visibility monitoring. Use this skill whenever the user asks about SEO, GEO, keyword research, SERP analysis, Google Search Console, technical audit, site audit, backlinks, link building, backlink analysis, AI visibility, AI Overviews, schema markup, structured data, Core Web Vitals, page speed, content gap analysis, content briefs, anchor text, referring domains, broken links, orphan pages, meta descriptions, title tags, H1 tags, canonical tags, sitemap, robots.txt, crawl budget, link graph, featured snippets, People Also Ask, related searches, search intent, keyword clustering, topic clusters, SERP features, AI audit fixes, Serper.dev, DataForSEO, Common Crawl, Google Trends, or any search engine optimisation and AI search visibility task. Google SERP scraping is DEAD — always use Serper.dev API. DataForSEO backlinks requires $100/month — us4---56## Reference Files — Load When Relevant78| Topic | File | Load when... |9|---|---|---|10| Keyword research stack | `references/keyword-research.md` | Finding keywords, clustering, search volume, intent |11| SERP analysis | `references/serp-analysis.md` | Analysing SERPs, AI Overviews, content gaps, briefs |12| Technical audit | `references/technical-audit.md` | Site audits, crawling, meta tags, schema, Core Web Vitals |13| Backlink intelligence | `references/backlinks.md` | Backlink discovery, toxic scoring, competitor gaps, broken links |14| Infrastructure | `references/infrastructure.md` | Building tools, proxies, caching, deployment, APIs |15| Failure registry | `references/failures.md` | Debugging, known pitfalls, API failures, library bugs |1617## Project Context1819If `project-context.md` exists in the skill directory or `.claude/` folder, read it20before any task. It contains brand details, website URLs, API credentials paths,21infrastructure setup, and content defaults — avoids re-explaining context each session.2223## Workflow Overview2425```261. Keyword Research → Find opportunities, cluster by intent, assess difficulty272. SERP Analysis → Analyse top results, track AI Overviews, generate content briefs283. Technical Audit → Crawl site, check on-page SEO, validate schema, measure Core Web Vitals294. Content Optimisation → Fill content gaps, optimise meta/headings/schema, improve AI extractability305. Backlink Intelligence → Discover links, score quality, find toxic links, analyse competitor gaps316. Monitoring → Track rankings, AI Overview rates, backlink changes, technical health32```3334## Step 1: Keyword Research3536### Free Stack (No Paid APIs Required)3738| Source | What it gives you | Cost |39|---|---|---|40| Google Autocomplete | Real-time query suggestions | Free (use Webshare proxies) |41| YouTube Autocomplete | Video-intent keywords | Free |42| People Also Ask | Question-format keywords | Free (via Serper SERP data) |43| Related Searches | Semantic variations | Free (via Serper SERP data) |44| GSC API | Queries you already rank for | Free (service account) |45| Serper Autocomplete | Google Trends proxy | Free (2,500/month) |46| DataForSEO | Volume, CPC, competition | Optional ($1 free credit) |4748### AI Clustering4950Send keywords to Groq (llama-3.1-8b-instant) — returns semantic topic clusters51with parent topics and sub-topics. Free tier, no credit card required.5253### Intent Classification5455Rule-based classifier — no API needed:56- **Transactional**: buy, price, discount, coupon, deal, order, shop57- **Commercial**: best, top, review, comparison, vs, alternative58- **Informational**: how, what, why, when, guide, tutorial, learn59- **Navigational**: brand name, login, dashboard, app, official60- **Local**: near me, in [city], directions, open now, hours6162### Proxy Difficulty Score63640–100 score calculated without paid APIs:65- Count strong domains in top 10 (Wikipedia, Reddit, .gov, .edu, major brands)66- Detect AI Overview presence (+15 difficulty)67- Detect featured snippet (+10 difficulty)68- Brand detection — owned domain queries get reduced difficulty6970See `references/keyword-research.md` for deep alphabet soup mode, monthly trends,71sparkline data, and the full implementation stack.7273## Step 2: SERP Analysis7475### Critical Rule7677**Google SERP scraping is dead.** Google returns JS-only noscript pages from any78IP, proxy, or user agent. Do not attempt to scrape google.com directly.7980**Use Serper.dev API instead:**81- 2,500 free searches/month82- `POST https://google.serper.dev/search`83- Returns: organic results, AI Overview, featured snippets, PAA, knowledge panel,84 videos, related searches — all structured JSON8586### AI Overview Tracking8788Daily cron job tracks AI Overview presence per keyword. Calculate:89- Overall AI Overview rate (percentage of tracked keywords with AO)90- Per-keyword AO history (appeared/disappeared over time)91- Content cited in AI Overviews (your domain vs competitors)9293### Content Gap Analysis9495For any target keyword:961. Pull top 5 SERP results via Serper972. Fetch and parse each page (cheerio)983. Compare: headings, word count, schema types, topics covered994. Generate gap report: what competitors cover that you don't100101### Content Brief Generation102103Auto-generate briefs from SERP data:104- Recommended word count (median of top 5)105- Must-have topics (present in 3+ of top 5)106- Gap topics (present in competitors, missing from your page)107- Schema suggestions based on SERP features108- Recommended headings structure109110See `references/serp-analysis.md` for SERP feature history tracking,111content brief templates, and implementation details.112113## Step 3: Technical Audit114115### Crawl Strategy116117Sitemap-first crawl:1181. Fetch `/sitemap.xml` (handle sitemap index with nested sub-sitemaps)1192. Extract all URLs1203. Fetch each page, parse with cheerio1214. Run per-page and site-wide checks122123### Per-Page Checks124125| Check | Pass Criteria |126|---|---|127| Title | Present, 30–60 characters |128| Meta description | Present, 120–160 characters |129| H1 | Exactly one per page |130| Canonical | Present, self-referencing or valid |131| Viewport | Present (`width=device-width`) |132| OG tags | og:title, og:description, og:image present |133| Schema | Valid JSON-LD with required fields |134| Images | All `<img>` have non-empty `alt` attributes |135| Word count | ≥300 for content pages |136| HTML size | Flag if >1 MB |137138### Site-Wide Checks139140- Duplicate titles across pages141- Duplicate meta descriptions across pages142- Orphan pages (no inbound internal links)143- Deep pages (>3 clicks from homepage)144145### Schema Validation146147JSON-LD required fields per type:148- **Article**: headline, datePublished, author, image149- **FAQ**: mainEntity with Question + acceptedAnswer150- **HowTo**: name, step[] with text151- **Product**: name, offers with price + priceCurrency152- **Organization**: name, url, logo153154### Core Web Vitals155156Google PSI API (free, no key required for basic use):157- LCP (Largest Contentful Paint) — target < 2.5s158- FID/INP (Interaction to Next Paint) — target < 200ms159- CLS (Cumulative Layout Shift) — target < 0.1160- FCP, TTFB, TBT as supplementary metrics161162### AI Audit Fix Suggestions163164Send failing checks to Groq (llama-3.1-8b-instant):165- Generate replacement titles at correct length166- Write meta descriptions from page content167- Suggest alt text from image context168- Flag thin content pages with expansion recommendations169170See `references/technical-audit.md` for cheerio bug workarounds,171link graph construction, and full implementation details.172173## Step 4: Content Optimisation174175### On-Page SEO Checklist176177- [ ] Primary keyword in title (front-loaded)178- [ ] Primary keyword in H1179- [ ] Primary keyword in first 100 words180- [ ] Secondary keywords in H2/H3 headings181- [ ] Internal links to related content (3–5 minimum)182- [ ] External links to authoritative sources (1–3)183- [ ] Schema markup matching content type184- [ ] Images with descriptive alt text containing keywords185- [ ] Meta description with CTA and primary keyword186- [ ] URL slug contains primary keyword (short, no stop words)187188### AI Visibility (GEO) Optimisation189190- Structure content with clear headings and bullet points191- Include direct, quotable answers to common questions192- Add FAQ schema for question-based queries193- Use statistics and citations that AI can extract194- Ensure schema markup is comprehensive and accurate195- Monitor AI Overview citations for your domain196197## Step 5: Backlink Intelligence198199### Free Discovery Stack200201| Method | What it finds | Limitation |202|---|---|---|203| Common Crawl CDX | Historical backlinks across 3+ indexes | Not real-time |204| Live HTTP verification | Current link status | Slow for large sets |205| Paste importers | Links from free tools (Moz, Neil Patel, etc.) | Manual paste step |206207### Quality Scoring208209Each backlink scored 0–100 with A–F grade:210- **Link location**: content (+40) > sidebar (+15) > footer (+5) > nav (+5)211- **Anchor quality**: keyword-rich (+20) > branded (+15) > generic (+5) > naked URL (+3)212- **Link density**: fewer outbound links on page = higher score213- **Page freshness**: recent content scores higher214- **Follow status**: dofollow (+15) > nofollow (+5)215216### Toxic Link Detection2172184-tier classification based on:219- Spam TLDs (.xyz, .top, .buzz, .click, .loan, .work)220- Numeric-heavy domains221- Excessive hyphens in domain222- Spam keywords in URL path (casino, pharmacy, payday, etc.)223224### Competitor Gap Analysis225226Common Crawl-based diff:2271. Query CC for domains linking to competitor2282. Query CC for domains linking to you2293. Diff = opportunity domains (link to them, not you)230231See `references/backlinks.md` for anchor diversity (Shannon entropy),232broken outbound link checking, and paste importer formats.233234## Step 6: Monitoring235236### Recommended Cron Schedule237238| Task | Frequency | Tool |239|---|---|---|240| Rank tracking (via Serper) | Daily | PM2 cron |241| AI Overview monitoring | Daily | PM2 cron |242| GSC data pull | Weekly | PM2 cron |243| Technical audit | Weekly | PM2 cron |244| Backlink discovery | Monthly | PM2 cron |245| Core Web Vitals | Monthly | PSI API |246247### Alert Triggers248249- Rank drop >5 positions for tracked keyword250- AI Overview appears/disappears for tracked keyword251- New backlink from high-authority domain252- Technical issue detected (broken page, missing schema)253- Core Web Vitals regression254255## Quick Decision Trees256257### "I need more organic traffic"258259```260Is your site technically sound?261├── No → Run Technical Audit (Step 3)262└── Yes263 ├── Do you have keyword targets?264 │ ├── No → Run Keyword Research (Step 1)265 │ └── Yes266 │ ├── Are you ranking for them?267 │ │ ├── No → Run SERP Analysis + Content Optimisation (Steps 2+4)268 │ │ └── Yes, but low positions269 │ │ ├── Content gap? → Fill gaps (Step 2)270 │ │ └── Authority gap? → Backlink Intelligence (Step 5)271 │ └── Are you visible in AI Overviews?272 │ ├── No → GEO Optimisation (Step 4)273 │ └── Yes → Monitor and maintain (Step 6)274```275276### "My rankings dropped"277278```279When did it drop?280├── After a Google update → Check Search Status Dashboard, wait 2 weeks281├── After site changes → Technical Audit (Step 3), check canonical/redirect issues282├── Gradual decline283│ ├── Content freshness issue → Update content, add new sections284│ ├── Competitors improved → SERP Analysis (Step 2), Content Gap285│ └── Lost backlinks → Backlink monitoring, re-acquisition outreach286```287288### "I want to track AI visibility"289290```2911. Set up daily AI Overview tracking via Serper (Step 2)2922. Monitor citation rate for your domain in AI Overviews2933. Optimise content for extractability (Step 4 — GEO)2944. Add comprehensive schema markup (Step 3)2955. Track changes weekly in monitoring dashboard (Step 6)296```