Web visibility (SEO + AEO/GEO)
How to get found by both classic search and AI answer engines. The space is full
of hype; this skill encodes what the research actually shows (academic papers +
official docs + vendor specs — sources at the end). Default to the evidence
hierarchy below before implementing anything.
First principle: credibility + structure beat tricks
The only peer-reviewed controlled study (GEO, KDD 2024) found the biggest gains
in AI-answer visibility came from quotations (+41%), statistics (+32%), inline
citations (+28%), and fluency (+28%) — and keyword stuffing measurably
hurt. Citations gave +115% for a page ranked #5: the leverage is largest
for newer, lower-authority sites. So: write credible, well-structured, quotable
content; never stuff keywords.
Evidence hierarchy — what to do vs. skip
| Mechanism |
Reality (2026) |
Call |
| schema.org / JSON-LD |
The ONLY machine-readable signal AI consumes at scale (Google + Bing→ChatGPT) |
DO — top priority |
| AEO/GEO content (quotes, stats, citations, front-loaded answers, freshness, semantic HTML) |
Peer-reviewed effect sizes; biggest lever for small sites |
DO |
| robots.txt (RFC 9309) + Content Signals |
Real crawler control; "block training, keep search"; voluntary |
DO |
Sitemap + honest lastmod + canonical + internal links |
Core, proven; priority/changefreq ignored by Google |
DO |
| Core Web Vitals (LCP ≤2.5s, INP ≤200ms, CLS ≤0.1, mobile-first) |
Quality tiebreaker, not a boost |
DO |
| IndexNow |
Bing/Yandex honor it (Google doesn't); Bing gates ChatGPT Search ~87% |
DO (for Bing) |
| llms.txt / llms-full.txt |
Vendors don't consume it (~97% of files never fetched); only coding agents fetch |
DO cheaply — as dev-agent convenience, NEVER sold as SEO |
Markdown twins (serve .md to agents via Accept) |
Emerging; near-free if content is already Markdown |
DO if cheap |
| security.txt (RFC 9116) |
IANA-permanent hygiene |
DO (trivial) |
| FAQPage / HowTo rich results |
Deprecated by Google (gone June 2026) |
Keep visible Q&A (aids AI citation); do NOT chase the markup |
| Google Indexing API |
Eligible only for JobPosting/livestream |
SKIP |
| aggregateRating without real reviews |
Manual-action risk |
SKIP until genuine on-page reviews |
| agents.json / ai-plugin.json / .well-known/mcp |
Niche/stalled/deprecated; not in IANA registry |
SKIP |
| Keyword stuffing / thin schema |
Proven to hurt / underperform |
AVOID |
| hreflang on a single-language site |
Adds risk, no benefit |
AVOID |
The architecture: one source of truth → every artifact
Never hand-author discoverability files. Generate page metadata, JSON-LD,
sitemap.xml, robots.txt, llms.txt, and Markdown twins from one content
model so a new page lights up everywhere and nothing drifts. Reference
implementation (FastAPI/Jinja): control-plane/seo.py in ever-just/ww.everjust.app.
JSON-LD recipes (the high-ROI part)
Place <script type="application/ld+json"> in <head>; markup must match
visible content (anti-spam). Validate every block (see Validation).
- Organization (homepage):
name, url, logo, sameAs (real social
profiles only), contactPoint.
- WebSite (homepage):
name, alternateName, url, and a SearchAction
potentialAction (sitelinks searchbox eligibility).
- BreadcrumbList (catalog/docs): ≥2
ListItems; last may omit item.
- SoftwareApplication / Product (app/pricing):
name, offers.price. Add
aggregateRating only with genuine on-page reviews.
- Article / TechArticle (blog/docs):
headline, image, datePublished,
dateModified (ISO 8601 w/ tz), author (Organization w/ name+url).
robots.txt + Content Signals
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /private
Sitemap: https://example.com/sitemap.xml
- A site that wants reach welcomes search + answer crawlers;
ai-train is
the owner's lever (yes for max brand spread; no to opt out of training).
- robots.txt is voluntary (RFC 9309); user-triggered fetchers ignore it and
at least one vendor was caught evading — use a WAF for anything you must stop.
- Keep the AI-crawler user-agent list current from
github.com/ai-robots-txt/ai.robots.txt (robots.json).
llms.txt (cheap, honestly framed)
llmstxt.org format: # H1 → > summary blockquote → ## Section link lists →
## Optional. Hosted at /llms.txt; /llms-full.txt inlines content. Ship it
for coding agents (Claude Code/Cursor do fetch it) — never present it as an SEO
lever.
IndexNow (fast Bing → ChatGPT)
Host https://host/{key}.txt, then POST changed URLs to
https://api.indexnow.org/indexnow on deploy ({host, key, keyLocation, urlList}). ~30 lines; Google ignores it, Bing/Yandex/Naver/Seznam honor it.
AEO/GEO content checklist (apply to every important page)
Validation (do not ship malformed markup)
- Parse every
<script type="application/ld+json"> block as JSON in tests
(catches template breakage). For richer checks use scrapinghub/extruct.
- Google Rich Results Test + Search Console rich-result reports.
- PageSpeed/CrUX for Core Web Vitals at p75 mobile.
Auditing an existing site (quick pass)
robots.txt / sitemap.xml present, honest, not blocking noindex pages?- Self-referencing canonical, HTTPS, 200s, shallow internal linking?
- JSON-LD present, valid, matching visible content (Organization/WebSite at
minimum; Breadcrumb/Article/Product where relevant)?
- Pages front-load answers, cite sources/stats, show freshness, chunk cleanly?
- In Bing's index? IndexNow wired?
- Core Web Vitals green on mobile?
- Not doing the "skip" list (FAQ-markup chasing, fake ratings, keyword stuffing)?
Building visibility into a product (e.g., a website builder)
Bake the above in by default per generated site: JSON-LD from the user's real
data, an AI-aware robots.txt with an owner training-toggle, per-site llms.txt,
IndexNow on the site's domain, and an in-editor AEO checklist (front-loaded
answer, headings, freshness, alt text) so non-technical owners publish citable
pages. Respect tenant isolation (build only from that owner's data) and their
training preference.
Field-tested techniques (proven in production on everjust.app)
These are the parts that worked end-to-end, with the gotchas that bit us.
IndexNow with a committed public key (no secret, no server access). An
IndexNow key is public by design — the engine verifies host ownership by
fetching https://<host>/<key>.txt — so commit it (a DEFAULT_INDEXNOW_KEY
constant, env-overridable), serve the key file unconditionally, and ping
api.indexnow.org on every deploy + a weekly cron. No GitHub secret or prod
.env edit required. Submission caps that matter: Bing Webmaster API
SubmitUrlbatch = 50 URLs/day; IndexNow = effectively uncapped; sitemap
submission (SubmitFeed) = uncapped. So push everything via IndexNow + the
sitemap, and use the Bing URL-API only for a small priority set.
The demand-surface page pattern (data-driven, one template, many pages). The
highest-leverage owned surface is unbranded intent pages generated from data you
already keep:
/alternatives/<tool> — invert each product's "replaces" list → one page per
competitor ("X alternative", "open source alternative to X")./open-source/<category> — a curated map of "open source " →
the app(s) that deliver it.
Each: front-loaded answer, links to the real product page, visible Q&A,
SoftwareApplication + BreadcrumbList JSON-LD, auto-added to sitemap + llms.txt.
Be honest (no fabricated superiority); if the product is open-source-based, say
so plainly (a true, differentiated angle) — and respect debranding (don't name an
upstream the product hides).
Prioritize with real keyword volume. Pull free directional volume from the
Bing Webmaster Keyword Research API (GetKeyword, ISO YYYY-MM-DD dates —
not /Date(ms)/). What it showed for a SaaS: "[category] software" and
"open source [category]" volume dwarf "[brand] alternative" on Bing. Caveats:
Bing is ~3–4% of search (Google ~10–30×, so cross-check Keyword Planner for
absolutes); and "agent/LLM search volume" is not publicly measurable — track
your citation share (Profound/Otterly-type tools) instead of query counts.
Ops gotcha that caused a 502 outage: if nginx upstreams are static
(server name:port;, resolved once at startup), a deploy that recreates the
app containers gives them new Docker IPs and nginx keeps proxying the dead
ones → 502 on every vhost, while container-direct health checks still pass.
Fix: restart nginx at the end of the deploy (after backends are recreated),
or use a resolver + variable proxy_pass to resolve at request time. Always
verify recovery through nginx (the public URL), not just the container.
Sources
Academic: GEO (KDD 2024) arxiv.org/abs/2311.09735 · GEO-16 citation study
arxiv.org/html/2509.10762v1 · GEO across engines arxiv.org/html/2509.08919v1.
Official: developers.google.com/search · schema.org · RFC 9309 · llmstxt.org ·
contentsignals.org · indexnow.org/documentation · RFC 9116. Data: Seer (SearchGPT
87% Bing) · Ahrefs (llms.txt 97% unfetched) · github.com/ai-robots-txt/ai.robots.txt.
Full hierarchy and rationale: ww.everjust.app/docs/VISIBILITY_PLAN.md.
1---2name: web-visibility3description: Make a site or product maximally findable by search engines AND AI answer engines (ChatGPT Search, Perplexity, Google AI Overviews). Use when building visibility into a product, auditing a site's SEO/AEO/GEO, or deciding which discoverability mechanism is worth implementing. Evidence-based: prioritizes structured data and proven content patterns, and explicitly skips the hype (llms.txt-as-SEO, FAQ rich results, Indexing API). Includes JSON-LD recipes, robots.txt/llms.txt/IndexNow generators, and a validation step.4---56# Web visibility (SEO + AEO/GEO)78How to get found by both classic search and AI answer engines. The space is full9of hype; this skill encodes what the research actually shows (academic papers +10official docs + vendor specs — sources at the end). **Default to the evidence11hierarchy below before implementing anything.**1213## First principle: credibility + structure beat tricks1415The only peer-reviewed controlled study (GEO, KDD 2024) found the biggest gains16in AI-answer visibility came from **quotations (+41%), statistics (+32%), inline17citations (+28%), and fluency (+28%)** — and **keyword stuffing measurably18*hurt*.** Citations gave **+115% for a page ranked #5**: the leverage is largest19for newer, lower-authority sites. So: write credible, well-structured, quotable20content; never stuff keywords.2122## Evidence hierarchy — what to do vs. skip2324| Mechanism | Reality (2026) | Call |25|---|---|---|26| **schema.org / JSON-LD** | The ONLY machine-readable signal AI consumes at scale (Google + Bing→ChatGPT) | **DO — top priority** |27| **AEO/GEO content** (quotes, stats, citations, front-loaded answers, freshness, semantic HTML) | Peer-reviewed effect sizes; biggest lever for small sites | **DO** |28| **robots.txt (RFC 9309) + Content Signals** | Real crawler control; "block training, keep search"; voluntary | **DO** |29| **Sitemap + honest `lastmod` + canonical + internal links** | Core, proven; `priority`/`changefreq` ignored by Google | **DO** |30| **Core Web Vitals** (LCP ≤2.5s, INP ≤200ms, CLS ≤0.1, mobile-first) | Quality tiebreaker, not a boost | **DO** |31| **IndexNow** | Bing/Yandex honor it (Google doesn't); **Bing gates ChatGPT Search ~87%** | **DO (for Bing)** |32| **llms.txt / llms-full.txt** | Vendors don't consume it (~97% of files never fetched); only coding agents fetch | **DO cheaply — as dev-agent convenience, NEVER sold as SEO** |33| **Markdown twins** (serve `.md` to agents via `Accept`) | Emerging; near-free if content is already Markdown | **DO if cheap** |34| **security.txt** (RFC 9116) | IANA-permanent hygiene | **DO (trivial)** |35| **FAQPage / HowTo rich results** | **Deprecated by Google (gone June 2026)** | **Keep visible Q&A (aids AI citation); do NOT chase the markup** |36| **Google Indexing API** | Eligible only for JobPosting/livestream | **SKIP** |37| **aggregateRating without real reviews** | Manual-action risk | **SKIP until genuine on-page reviews** |38| **agents.json / ai-plugin.json / .well-known/mcp** | Niche/stalled/deprecated; not in IANA registry | **SKIP** |39| **Keyword stuffing / thin schema** | Proven to hurt / underperform | **AVOID** |40| **hreflang on a single-language site** | Adds risk, no benefit | **AVOID** |4142## The architecture: one source of truth → every artifact4344Never hand-author discoverability files. Generate page metadata, JSON-LD,45`sitemap.xml`, `robots.txt`, `llms.txt`, and Markdown twins from one content46model so a new page lights up everywhere and nothing drifts. Reference47implementation (FastAPI/Jinja): `control-plane/seo.py` in `ever-just/ww.everjust.app`.4849## JSON-LD recipes (the high-ROI part)5051Place `<script type="application/ld+json">` in `<head>`; markup must match52visible content (anti-spam). **Validate every block** (see Validation).5354- **Organization** (homepage): `name`, `url`, `logo`, `sameAs` (real social55 profiles only), `contactPoint`.56- **WebSite** (homepage): `name`, `alternateName`, `url`, and a `SearchAction`57 `potentialAction` (sitelinks searchbox eligibility).58- **BreadcrumbList** (catalog/docs): ≥2 `ListItem`s; last may omit `item`.59- **SoftwareApplication / Product** (app/pricing): `name`, `offers.price`. Add60 `aggregateRating` **only** with genuine on-page reviews.61- **Article / TechArticle** (blog/docs): `headline`, `image`, `datePublished`,62 `dateModified` (ISO 8601 w/ tz), `author` (Organization w/ `name`+`url`).6364## robots.txt + Content Signals6566```67User-agent: *68Content-Signal: search=yes, ai-input=yes, ai-train=yes69Allow: /70Disallow: /private71Sitemap: https://example.com/sitemap.xml72```7374- A site that wants reach **welcomes** search + answer crawlers; `ai-train` is75 the owner's lever (yes for max brand spread; no to opt out of training).76- robots.txt is **voluntary** (RFC 9309); user-triggered fetchers ignore it and77 at least one vendor was caught evading — use a WAF for anything you must stop.78- Keep the AI-crawler user-agent list current from79 `github.com/ai-robots-txt/ai.robots.txt` (`robots.json`).8081## llms.txt (cheap, honestly framed)8283llmstxt.org format: `# H1` → `> summary` blockquote → `## Section` link lists →84`## Optional`. Hosted at `/llms.txt`; `/llms-full.txt` inlines content. Ship it85for coding agents (Claude Code/Cursor do fetch it) — **never present it as an SEO86lever.**8788## IndexNow (fast Bing → ChatGPT)8990Host `https://host/{key}.txt`, then POST changed URLs to91`https://api.indexnow.org/indexnow` on deploy (`{host, key, keyLocation,92urlList}`). ~30 lines; Google ignores it, Bing/Yandex/Naver/Seznam honor it.9394## AEO/GEO content checklist (apply to every important page)9596- [ ] **Front-load a self-contained direct answer** to the page's core question97 in the first ~150 words (passage extraction + citation-position bias).98- [ ] **Cite credible sources inline**, use **concrete statistics with sources**,99 and **quote authorities** where truthful — the three biggest levers.100- [ ] **Show freshness**: visible "last updated" date; review quarterly.101- [ ] **Clean semantic HTML / heading hierarchy** so engines chunk cleanly.102- [ ] **Be in Bing's index** (gates ChatGPT Search) and Google's.103- [ ] **Earn third-party coverage** (answer engines favor earned over owned).104- [ ] **Never keyword-stuff; never fabricate** stats/quotes.105106## Validation (do not ship malformed markup)107108- Parse every `<script type="application/ld+json">` block as JSON in tests109 (catches template breakage). For richer checks use `scrapinghub/extruct`.110- Google Rich Results Test + Search Console rich-result reports.111- PageSpeed/CrUX for Core Web Vitals at p75 mobile.112113## Auditing an existing site (quick pass)1141151. `robots.txt` / `sitemap.xml` present, honest, not blocking `noindex` pages?1162. Self-referencing canonical, HTTPS, 200s, shallow internal linking?1173. JSON-LD present, valid, matching visible content (Organization/WebSite at118 minimum; Breadcrumb/Article/Product where relevant)?1194. Pages front-load answers, cite sources/stats, show freshness, chunk cleanly?1205. In Bing's index? IndexNow wired?1216. Core Web Vitals green on mobile?1227. Not doing the "skip" list (FAQ-markup chasing, fake ratings, keyword stuffing)?123124## Building visibility into a product (e.g., a website builder)125126Bake the above in **by default** per generated site: JSON-LD from the user's real127data, an AI-aware `robots.txt` with an owner training-toggle, per-site `llms.txt`,128IndexNow on the site's domain, and an in-editor AEO checklist (front-loaded129answer, headings, freshness, alt text) so non-technical owners publish citable130pages. Respect tenant isolation (build only from that owner's data) and their131training preference.132133## Field-tested techniques (proven in production on everjust.app)134135These are the parts that worked end-to-end, with the gotchas that bit us.136137**IndexNow with a committed public key (no secret, no server access).** An138IndexNow key is *public by design* — the engine verifies host ownership by139fetching `https://<host>/<key>.txt` — so commit it (a `DEFAULT_INDEXNOW_KEY`140constant, env-overridable), serve the key file unconditionally, and ping141`api.indexnow.org` on every deploy + a weekly cron. No GitHub secret or prod142`.env` edit required. Submission caps that matter: **Bing Webmaster API143`SubmitUrlbatch` = 50 URLs/day; IndexNow = effectively uncapped; sitemap144submission (`SubmitFeed`) = uncapped.** So push everything via IndexNow + the145sitemap, and use the Bing URL-API only for a small priority set.146147**The demand-surface page pattern (data-driven, one template, many pages).** The148highest-leverage owned surface is unbranded intent pages generated from data you149already keep:150- `/alternatives/<tool>` — invert each product's "replaces" list → one page per151 competitor ("X alternative", "open source alternative to X").152- `/open-source/<category>` — a curated map of "open source <category>" →153 the app(s) that deliver it.154Each: front-loaded answer, links to the real product page, visible Q&A,155`SoftwareApplication` + `BreadcrumbList` JSON-LD, auto-added to sitemap + llms.txt.156Be honest (no fabricated superiority); if the product is open-source-based, say157so plainly (a true, differentiated angle) — and respect debranding (don't name an158upstream the product hides).159160**Prioritize with real keyword volume.** Pull free directional volume from the161**Bing Webmaster Keyword Research API** (`GetKeyword`, ISO `YYYY-MM-DD` dates —162*not* `/Date(ms)/`). What it showed for a SaaS: **"[category] software" and163"open source [category]" volume dwarf "[brand] alternative"** on Bing. Caveats:164Bing is ~3–4% of search (Google ~10–30×, so cross-check Keyword Planner for165absolutes); and **"agent/LLM search volume" is not publicly measurable** — track166your *citation share* (Profound/Otterly-type tools) instead of query counts.167168**Ops gotcha that caused a 502 outage:** if nginx upstreams are static169(`server name:port;`, resolved once at startup), a deploy that *recreates* the170app containers gives them new Docker IPs and nginx keeps proxying the dead171ones → 502 on every vhost, while container-direct health checks still pass.172**Fix: restart nginx at the end of the deploy** (after backends are recreated),173or use a `resolver` + variable `proxy_pass` to resolve at request time. Always174verify recovery through nginx (the public URL), not just the container.175176## Sources177178Academic: GEO (KDD 2024) arxiv.org/abs/2311.09735 · GEO-16 citation study179arxiv.org/html/2509.10762v1 · GEO across engines arxiv.org/html/2509.08919v1.180Official: developers.google.com/search · schema.org · RFC 9309 · llmstxt.org ·181contentsignals.org · indexnow.org/documentation · RFC 9116. Data: Seer (SearchGPT18287% Bing) · Ahrefs (llms.txt 97% unfetched) · github.com/ai-robots-txt/ai.robots.txt.183Full hierarchy and rationale: `ww.everjust.app/docs/VISIBILITY_PLAN.md`.
ever-just