Agentic AI Optimization (AAIO)
Make sites visible and usable to AI crawlers, citation engines, and browser agents.
Based on Joel Hooks' implementation checklist. Not just "AI SEO" — it's the overlap of traditional SEO, static truth in initial HTML, machine-readable surfaces, and UX ergonomics for agents.
Core Principle
If AI crawlers can't read your pages and browser agents can't operate your UI, your site is invisible in the workflows that matter next.
8-Step Checklist
1. Crawl Policy
Set explicit robots.txt for AI bots. Three classes:
- Search/indexing: OAI-SearchBot, Googlebot, Bingbot, Claude-SearchBot, PerplexityBot → Allow
- User-triggered: ChatGPT-User, Claude-User, Perplexity-User → Allow
- Training: GPTBot, Google-Extended, ClaudeBot → Block (unless you want training)
Add sitemap. Use noindex on private pages. Don't confuse llms.txt with discoverability — it's a hint surface, not the main channel.
See references/robots-txt-template.md for copy-paste template.
2. Critical Content in Initial HTML
- Server-render every page you want cited (docs, pricing, features, FAQs)
- Don't ship blank SPA shells — if
curl can't see it, AI can't see it
- Don't hide facts behind tabs, modals, accordions, or client-side fetches
- Use real
<a href> links, not div + onClick
Verify: curl -sL URL | grep 'key fact' — if missing, fix rendering.
3. Structured Content Patterns
- One heading = one idea. One paragraph = one claim.
- Front-load answers in first sentence under each heading
- Sections must make sense when copied out of context
- Use Q&A blocks, numbered steps, bullets, tables for citable facts
- Replace vague adjectives with measurable claims
- Put canonical answer in ONE place — duplication creates conflict
4. JSON-LD Schema
Match schema to page type. See references/json-ld-templates.md for templates.
| Page Type |
Schema Types |
Minimum Fields |
| Site-wide |
Organization, WebSite |
name, url, logo |
| Article/doc |
Article, WebPage, BreadcrumbList |
headline, description, dates, author, publisher |
| FAQ |
FAQPage |
mainEntity[] with Question + acceptedAnswer |
| How-to |
HowTo |
name, step[] |
| Product |
Product, Offer |
name, description, price, availability, brand |
Rule: Every JSON-LD value must match visible page content exactly. Re-validate after every content change.
5. Accessibility Tree = Agent Interface
OpenAI Atlas, Microsoft browser automation, and most agent frameworks use ARIA roles/labels/states. A11y work IS agent-interface work.
- Native elements first:
<button>, <a>, <input>, <select>, <table>
- Label every form control. Add
autocomplete values.
- Use landmarks:
<nav>, <main>, <header>, <footer>
- Logical heading hierarchy. Descriptive link text.
- Expose state changes via ARIA.
Anti-patterns: <div class="button" onclick>, <input placeholder="Email"> without label, <div>
Verify: If Playwright getByRole() can't find it, agents can't either.
6. Machine Interfaces & Markdown Twins
The money section. A page can be indexable and still suck for agents.
Three-surface pattern:
/page → HTML for humans/page.md → Markdown for agents (Content-Type: text/markdown)/api/... → Structured JSON
Same canonical source. Different projections. No drift.
Discovery surfaces:
robots.txt → advertise sitemap.xml AND sitemap.mdsitemap.md → list human URLs + markdown twinsllms.txt → point to markdown sitemap and access patterns/api → discovery route with next_actions
MIME discipline:
- HTML:
text/html; charset=utf-8
- Markdown:
text/markdown; charset=utf-8
- JSON:
application/json; charset=utf-8
- llms.txt:
text/plain; charset=utf-8
If a markdown endpoint returns text/html, that's a bug.
See references/machine-interfaces.md for implementation patterns.
7. Agent-Ready Checkout (Commerce Only)
Skip for non-commerce. For e-commerce:
- Clean product catalog (precise titles, current prices, stable SKUs)
- Product + Offer schema on every product page
- Price/availability/shipping in visible HTML
- Evaluate Stripe Agentic Commerce Protocol if applicable
8. Measurement & Regression
- Track AI referrals separately (utm_source=chatgpt.com etc.)
- Log bot hits by user-agent
- Track citation presence for core queries
- Regression tests for important pages:
# AI crawler traffic
rg 'OAI-SearchBot|Googlebot|Claude-SearchBot|PerplexityBot' access.log
# Schema still present
curl -sL URL | rg 'application/ld\+json'
# Text-only smoke test
lynx -dump URL
Quick Audit Workflow
When auditing an existing site:
curl -s SITE/robots.txt — check AI bot policycurl -sL SITE/ | head -100 — facts in initial HTML?curl -sL SITE/ | rg 'application/ld\+json' — schema present?- Check heading hierarchy and content structure
- Test a11y tree in devtools or Playwright
- Check for markdown/API discovery surfaces
curl -I -A 'OAI-SearchBot/1.3' SITE/ — bot response correct?
For Coding Repos (AGENTS.md)
Passive always-on context in AGENTS.md beats optional skills the agent may never load:
## Agent retrieval hints
- Prefer retrieval-led reasoning over pretrained guesses
- Start with `/api`, `sitemap.md`, and `/.md` twins before scraping HTML
- Verify Content-Type before parsing
- Treat HTML, markdown, JSON as projections of same resource
## Operator path
- Machine-readable content: try `{page}.md` first
- Structured discovery: `/api`
- Broad site discovery: `/sitemap.md`
Definition of Done
1---2name: aaio3description: Agentic AI Optimization — make websites crawlable, citable, and usable by AI agents and browser automation. Use when building or auditing web projects for AI discoverability, implementing robots.txt AI bot policies, adding JSON-LD structured data, creating markdown twin routes, building accessibility trees for agent UX, or optimizing content structure for AI citation. Triggers on: AI SEO, AAIO, agentic optimization, AI discoverability, llms.txt, markdown twins, agent-ready, AI crawlers, structured data audit, schema.org, AI bot policy.4---56# Agentic AI Optimization (AAIO)78Make sites visible and usable to AI crawlers, citation engines, and browser agents.9Based on Joel Hooks' implementation checklist. Not just "AI SEO" — it's the overlap of traditional SEO, static truth in initial HTML, machine-readable surfaces, and UX ergonomics for agents.1011## Core Principle1213> If AI crawlers can't read your pages and browser agents can't operate your UI, your site is invisible in the workflows that matter next.1415## 8-Step Checklist1617### 1. Crawl Policy18Set explicit robots.txt for AI bots. Three classes:19- **Search/indexing:** OAI-SearchBot, Googlebot, Bingbot, Claude-SearchBot, PerplexityBot → Allow20- **User-triggered:** ChatGPT-User, Claude-User, Perplexity-User → Allow21- **Training:** GPTBot, Google-Extended, ClaudeBot → Block (unless you want training)2223Add sitemap. Use `noindex` on private pages. Don't confuse `llms.txt` with discoverability — it's a hint surface, not the main channel.2425See `references/robots-txt-template.md` for copy-paste template.2627### 2. Critical Content in Initial HTML28- Server-render every page you want cited (docs, pricing, features, FAQs)29- Don't ship blank SPA shells — if `curl` can't see it, AI can't see it30- Don't hide facts behind tabs, modals, accordions, or client-side fetches31- Use real `<a href>` links, not `div + onClick`3233**Verify:** `curl -sL URL | grep 'key fact'` — if missing, fix rendering.3435### 3. Structured Content Patterns36- One heading = one idea. One paragraph = one claim.37- Front-load answers in first sentence under each heading38- Sections must make sense when copied out of context39- Use Q&A blocks, numbered steps, bullets, tables for citable facts40- Replace vague adjectives with measurable claims41- Put canonical answer in ONE place — duplication creates conflict4243### 4. JSON-LD Schema44Match schema to page type. See `references/json-ld-templates.md` for templates.4546| Page Type | Schema Types | Minimum Fields |47|-----------|-------------|----------------|48| Site-wide | Organization, WebSite | name, url, logo |49| Article/doc | Article, WebPage, BreadcrumbList | headline, description, dates, author, publisher |50| FAQ | FAQPage | mainEntity[] with Question + acceptedAnswer |51| How-to | HowTo | name, step[] |52| Product | Product, Offer | name, description, price, availability, brand |5354**Rule:** Every JSON-LD value must match visible page content exactly. Re-validate after every content change.5556### 5. Accessibility Tree = Agent Interface57OpenAI Atlas, Microsoft browser automation, and most agent frameworks use ARIA roles/labels/states. A11y work IS agent-interface work.5859- Native elements first: `<button>`, `<a>`, `<input>`, `<select>`, `<table>`60- Label every form control. Add `autocomplete` values.61- Use landmarks: `<nav>`, `<main>`, `<header>`, `<footer>`62- Logical heading hierarchy. Descriptive link text.63- Expose state changes via ARIA.6465**Anti-patterns:** `<div class="button" onclick>`, `<input placeholder="Email">` without label, `<div onclick="location.href">`6667**Verify:** If Playwright `getByRole()` can't find it, agents can't either.6869### 6. Machine Interfaces & Markdown Twins70The money section. A page can be indexable and still suck for agents.7172**Three-surface pattern:**73- `/page` → HTML for humans74- `/page.md` → Markdown for agents (Content-Type: `text/markdown`)75- `/api/...` → Structured JSON7677Same canonical source. Different projections. No drift.7879**Discovery surfaces:**80- `robots.txt` → advertise sitemap.xml AND sitemap.md81- `sitemap.md` → list human URLs + markdown twins82- `llms.txt` → point to markdown sitemap and access patterns83- `/api` → discovery route with `next_actions`8485**MIME discipline:**86- HTML: `text/html; charset=utf-8`87- Markdown: `text/markdown; charset=utf-8`88- JSON: `application/json; charset=utf-8`89- llms.txt: `text/plain; charset=utf-8`9091If a markdown endpoint returns `text/html`, that's a bug.9293See `references/machine-interfaces.md` for implementation patterns.9495### 7. Agent-Ready Checkout (Commerce Only)96Skip for non-commerce. For e-commerce:97- Clean product catalog (precise titles, current prices, stable SKUs)98- Product + Offer schema on every product page99- Price/availability/shipping in visible HTML100- Evaluate Stripe Agentic Commerce Protocol if applicable101102### 8. Measurement & Regression103- Track AI referrals separately (utm_source=chatgpt.com etc.)104- Log bot hits by user-agent105- Track citation presence for core queries106- Regression tests for important pages:107108```bash109# AI crawler traffic110rg 'OAI-SearchBot|Googlebot|Claude-SearchBot|PerplexityBot' access.log111# Schema still present112curl -sL URL | rg 'application/ld\+json'113# Text-only smoke test114lynx -dump URL115```116117## Quick Audit Workflow118When auditing an existing site:1191. `curl -s SITE/robots.txt` — check AI bot policy1202. `curl -sL SITE/ | head -100` — facts in initial HTML?1213. `curl -sL SITE/ | rg 'application/ld\+json'` — schema present?1224. Check heading hierarchy and content structure1235. Test a11y tree in devtools or Playwright1246. Check for markdown/API discovery surfaces1257. `curl -I -A 'OAI-SearchBot/1.3' SITE/` — bot response correct?126127## For Coding Repos (AGENTS.md)128Passive always-on context in AGENTS.md beats optional skills the agent may never load:129130```markdown131## Agent retrieval hints132- Prefer retrieval-led reasoning over pretrained guesses133- Start with `/api`, `sitemap.md`, and `/.md` twins before scraping HTML134- Verify Content-Type before parsing135- Treat HTML, markdown, JSON as projections of same resource136137## Operator path138- Machine-readable content: try `{page}.md` first139- Structured discovery: `/api`140- Broad site discovery: `/sitemap.md`141```142143## Definition of Done144- [ ] Search bots allowed, training bots blocked145- [ ] Critical facts in raw HTML146- [ ] Descriptive headings, direct answers, lists, tables147- [ ] JSON-LD validates148- [ ] A11y tree exposes buttons, links, forms, state149- [ ] Markdown/JSON discovery surfaces exist150- [ ] Analytics tracking AI referrals
KevinZai