AI Search Visibility
Separate "may my content train models?" from "may AI search tools find and cite me?", then configure each crawler on purpose.
There is no verified ranking formula for AI answers. This skill covers the parts you control and can verify: crawler access, indexability, and clear, quotable content.
When to Use
- Choosing a robots.txt policy for AI crawlers
- "We blocked AI bots and disappeared from ChatGPT/Perplexity search" (or the reverse)
- Evaluating whether to publish
llms.txt
- Making docs or product pages easier for answer engines to quote correctly
When NOT to Use
- General crawl/index problems in Google Search (use
technical-seo-audit)
- Legal questions about copyright or licensing; this skill describes vendor-documented controls only
Workflow
Step 1: Decide the policy per purpose
Most vendors now separate crawlers by purpose. Agree on three decisions with the site owner:
| Purpose |
Question |
| Model training |
May future models be trained on this content? |
| AI search / indexing |
May AI search products index and link to this content? |
| User-triggered fetch |
When a user asks an assistant to read a URL, may it fetch the page? |
Blocking training does not have to mean blocking search, and vice versa.
Step 2: Map purposes to documented user agents
As checked on 2026-11-04 from each vendor's official page. Vendors add and rename agents; re-verify before shipping.
| Vendor |
Token |
Purpose (per vendor docs) |
robots.txt |
| OpenAI |
GPTBot |
Content that may be used to train generative AI foundation models |
Honored; disallow signals "do not use for training" |
| OpenAI |
OAI-SearchBot |
Surfacing sites in ChatGPT search features |
Honored; disallow removes you from ChatGPT search answers |
| OpenAI |
ChatGPT-User |
User actions in ChatGPT / custom GPTs |
OpenAI states robots.txt rules may not apply to user-initiated requests |
| OpenAI |
OAI-AdsBot |
Checking ad landing pages; OpenAI says data is not used for training |
See OpenAI docs |
| Anthropic |
ClaudeBot |
Collecting content that may contribute to model training |
Disallow excludes future site content from training datasets |
| Anthropic |
Claude-SearchBot |
Indexing to improve search result quality |
Disallow prevents indexing for search |
| Anthropic |
Claude-User |
Fetching pages when a Claude user asks |
Disallow prevents retrieval in response to user queries |
| Google |
Google-Extended |
Product token: Gemini training and grounding in Gemini Apps / Vertex AI |
Does not affect Google Search inclusion or ranking; not a separate crawler |
| Perplexity |
PerplexityBot |
Surfacing and linking sites in Perplexity search results |
Honored |
| Perplexity |
Perplexity-User |
Fetching pages for user requests |
Perplexity states it generally ignores robots.txt |
| Apple |
Applebot |
Spotlight, Siri, Safari search features |
Honored |
| Apple |
Applebot-Extended |
Opt out of training Apple's foundation models |
Pages can still appear in Apple search results |
| Common Crawl |
CCBot |
Open web crawl used by many downstream datasets |
Honored via Disallow |
Key consequences:
- Google AI Overviews and AI Mode are part of Google Search.
Google-Extended does not control them. To limit how Search content appears there, Google points to nosnippet, data-nosnippet, max-snippet, or noindex, which also affect regular Search snippets.
- User-triggered fetchers may not obey robots.txt (OpenAI and Perplexity say so explicitly). Use authentication if content must not be fetched.
- Anthropic notes that IP blocking may not work as a reliable opt-out because it also blocks reading robots.txt.
Step 3: Write the robots.txt
A crawler follows only the most specific group that matches it, so give each AI agent its own group rather than relying on User-agent: *.
Example: allow AI search and user fetches, opt out of training.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/
User-agent: *
Disallow: /account/
Sitemap: https://www.example.com/sitemap.xml
Notes:
- Multiple
User-agent lines may share one group (RFC 9309).
Allow: / plus a narrower Disallow works because the longest matching rule wins.
Crawl-delay is non-standard. Anthropic documents support for it; Google does not support it. Others: check their docs.
- Disallowing
CCBot affects future Common Crawl snapshots, not copies already published.
Verify with the live file:
curl -s https://www.example.com/robots.txt
Step 4: Make sure search-oriented AI can actually use the page
AI search features generally build on ordinary crawling and indexing. For Google specifically, a page must be indexed and eligible for a snippet to be a supporting link in AI Overviews or AI Mode, and Google states there are no extra requirements, no special schema, and no "AI text files" needed.
So the checklist is the normal one:
- 200 status, indexable, not blocked, canonical set correctly
- Main content in server-rendered HTML (not every crawler runs JavaScript)
- Accurate
<title>, meta description, and headings
Step 5: Write content that is easy to quote correctly
These are editorial practices, not ranking guarantees:
- State the answer plainly near the top, then the detail. One claim per sentence where accuracy matters (prices, limits, versions, dates).
- Put facts in text, not only in images or interactive widgets.
- Date-stamp facts that change ("as of 2026-11-04") and keep them current.
- Use descriptive headings that match how people phrase questions.
- Cite primary sources for claims.
- Add JSON-LD that matches visible content (
Organization, Product, Article, SoftwareApplication, BreadcrumbList). Structured data helps machines parse the page; it does not guarantee inclusion anywhere.
Step 6: llms.txt (optional, low priority)
Facts:
llms.txt is a proposal by Jeremy Howard (published September 2024 at llmstxt.org), not an IETF or W3C standard.
- Format: a markdown file at
/llms.txt with a required H1 (site name), an optional blockquote summary, and optional H2 sections listing links. The proposal also suggests serving markdown versions of pages (page.md or page.html.md).
- Google's AI features documentation says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode.
- As of 2026-11-04 we found no documentation from Google, OpenAI, Anthropic, Perplexity, or Apple stating that their search crawlers use
llms.txt for ranking or inclusion. Treat claims otherwise as unverified.
When it is still worth doing: developer documentation, where users paste docs into coding assistants or agent tools fetch them directly. It is cheap to generate from your docs index.
# Example Docs
> Example is an open-source job queue for Node.js.
## Docs
- [Quickstart](https://example.com/docs/quickstart.md): install and run a first job
- [API reference](https://example.com/docs/api.md): every public method
## Optional
- [Changelog](https://example.com/changelog.md)
Generate it from the same source as the sitemap so it never drifts.
Step 7: Measure honestly
- Server logs: count requests by user agent and status code. Verify identity where vendors publish IP ranges; user-agent strings can be spoofed.
- Referral traffic from AI products appears in analytics only when the product passes a referrer or UTM; absence of referrals is not proof of absence.
- Do not report "AI visibility scores" from third-party tools as fact; describe their method.
Best Practices
- Decide training, search, and user-fetch policy separately, in writing.
- Name each AI agent explicitly; do not rely on the
* group.
- Re-check vendor crawler pages before each change; names and behaviors change.
- Use authentication, not robots.txt, for content that must not be fetched.
- Fix ordinary indexability first; AI search builds on it.
- Keep factual claims dated and sourced so quotes stay accurate.
- Treat llms.txt as optional convenience, not a visibility lever.
References
Checked 2026-11-04 (crawler lists change often; re-verify):
1---2name: ai-search-visibility3description: |4 Decide and implement how a site is accessed by AI crawlers, AI search features, and answer engines: which user agents to allow or block in robots.txt (training vs search vs user-triggered fetchers), what Google's AI features actually require, an honest take on llms.txt, and content/structured-data practices that make pages easy to quote accurately.5 Use this skill when someone asks how to appear in ChatGPT search, Claude, Perplexity, Google AI Overviews or AI Mode, wants to block AI training without losing search visibility, or asks whether they need llms.txt.6 Activate when: AI search, answer engine optimization, AEO, GEO, generative engine optimization, AI Overviews, AI Mode, GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, llms.txt, block AI crawlers.7metadata:8 source: https://skillmd.com/skills/latestaiagents/ai-search-visibility9---1011# AI Search Visibility1213**Separate "may my content train models?" from "may AI search tools find and cite me?", then configure each crawler on purpose.**1415There is no verified ranking formula for AI answers. This skill covers the parts you control and can verify: crawler access, indexability, and clear, quotable content.1617## When to Use1819- Choosing a robots.txt policy for AI crawlers20- "We blocked AI bots and disappeared from ChatGPT/Perplexity search" (or the reverse)21- Evaluating whether to publish `llms.txt`22- Making docs or product pages easier for answer engines to quote correctly2324## When NOT to Use2526- General crawl/index problems in Google Search (use `technical-seo-audit`)27- Legal questions about copyright or licensing; this skill describes vendor-documented controls only2829## Workflow3031### Step 1: Decide the policy per purpose3233Most vendors now separate crawlers by purpose. Agree on three decisions with the site owner:3435| Purpose | Question |36|---------|----------|37| Model training | May future models be trained on this content? |38| AI search / indexing | May AI search products index and link to this content? |39| User-triggered fetch | When a user asks an assistant to read a URL, may it fetch the page? |4041Blocking training does not have to mean blocking search, and vice versa.4243### Step 2: Map purposes to documented user agents4445As checked on 2026-11-04 from each vendor's official page. Vendors add and rename agents; re-verify before shipping.4647| Vendor | Token | Purpose (per vendor docs) | robots.txt |48|--------|-------|---------------------------|------------|49| OpenAI | `GPTBot` | Content that may be used to train generative AI foundation models | Honored; disallow signals "do not use for training" |50| OpenAI | `OAI-SearchBot` | Surfacing sites in ChatGPT search features | Honored; disallow removes you from ChatGPT search answers |51| OpenAI | `ChatGPT-User` | User actions in ChatGPT / custom GPTs | OpenAI states robots.txt rules may not apply to user-initiated requests |52| OpenAI | `OAI-AdsBot` | Checking ad landing pages; OpenAI says data is not used for training | See OpenAI docs |53| Anthropic | `ClaudeBot` | Collecting content that may contribute to model training | Disallow excludes future site content from training datasets |54| Anthropic | `Claude-SearchBot` | Indexing to improve search result quality | Disallow prevents indexing for search |55| Anthropic | `Claude-User` | Fetching pages when a Claude user asks | Disallow prevents retrieval in response to user queries |56| Google | `Google-Extended` | Product token: Gemini training and grounding in Gemini Apps / Vertex AI | Does not affect Google Search inclusion or ranking; not a separate crawler |57| Perplexity | `PerplexityBot` | Surfacing and linking sites in Perplexity search results | Honored |58| Perplexity | `Perplexity-User` | Fetching pages for user requests | Perplexity states it generally ignores robots.txt |59| Apple | `Applebot` | Spotlight, Siri, Safari search features | Honored |60| Apple | `Applebot-Extended` | Opt out of training Apple's foundation models | Pages can still appear in Apple search results |61| Common Crawl | `CCBot` | Open web crawl used by many downstream datasets | Honored via `Disallow` |6263Key consequences:6465- Google AI Overviews and AI Mode are part of Google Search. `Google-Extended` does not control them. To limit how Search content appears there, Google points to `nosnippet`, `data-nosnippet`, `max-snippet`, or `noindex`, which also affect regular Search snippets.66- User-triggered fetchers may not obey robots.txt (OpenAI and Perplexity say so explicitly). Use authentication if content must not be fetched.67- Anthropic notes that IP blocking may not work as a reliable opt-out because it also blocks reading robots.txt.6869### Step 3: Write the robots.txt7071A crawler follows only the most specific group that matches it, so give each AI agent its own group rather than relying on `User-agent: *`.7273Example: allow AI search and user fetches, opt out of training.7475```text76User-agent: GPTBot77Disallow: /7879User-agent: ClaudeBot80Disallow: /8182User-agent: Google-Extended83Disallow: /8485User-agent: Applebot-Extended86Disallow: /8788User-agent: CCBot89Disallow: /9091User-agent: OAI-SearchBot92User-agent: Claude-SearchBot93User-agent: PerplexityBot94Allow: /95Disallow: /account/9697User-agent: *98Disallow: /account/99100Sitemap: https://www.example.com/sitemap.xml101```102103Notes:104105- Multiple `User-agent` lines may share one group (RFC 9309).106- `Allow: /` plus a narrower `Disallow` works because the longest matching rule wins.107- `Crawl-delay` is non-standard. Anthropic documents support for it; Google does not support it. Others: check their docs.108- Disallowing `CCBot` affects future Common Crawl snapshots, not copies already published.109110Verify with the live file:111112```bash113curl -s https://www.example.com/robots.txt114```115116### Step 4: Make sure search-oriented AI can actually use the page117118AI search features generally build on ordinary crawling and indexing. For Google specifically, a page must be indexed and eligible for a snippet to be a supporting link in AI Overviews or AI Mode, and Google states there are no extra requirements, no special schema, and no "AI text files" needed.119120So the checklist is the normal one:121122- 200 status, indexable, not blocked, canonical set correctly123- Main content in server-rendered HTML (not every crawler runs JavaScript)124- Accurate `<title>`, meta description, and headings125126### Step 5: Write content that is easy to quote correctly127128These are editorial practices, not ranking guarantees:129130- State the answer plainly near the top, then the detail. One claim per sentence where accuracy matters (prices, limits, versions, dates).131- Put facts in text, not only in images or interactive widgets.132- Date-stamp facts that change ("as of 2026-11-04") and keep them current.133- Use descriptive headings that match how people phrase questions.134- Cite primary sources for claims.135- Add JSON-LD that matches visible content (`Organization`, `Product`, `Article`, `SoftwareApplication`, `BreadcrumbList`). Structured data helps machines parse the page; it does not guarantee inclusion anywhere.136137### Step 6: llms.txt (optional, low priority)138139Facts:140141- `llms.txt` is a proposal by Jeremy Howard (published September 2024 at llmstxt.org), not an IETF or W3C standard.142- Format: a markdown file at `/llms.txt` with a required H1 (site name), an optional blockquote summary, and optional H2 sections listing links. The proposal also suggests serving markdown versions of pages (`page.md` or `page.html.md`).143- Google's AI features documentation says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode.144- As of 2026-11-04 we found no documentation from Google, OpenAI, Anthropic, Perplexity, or Apple stating that their search crawlers use `llms.txt` for ranking or inclusion. Treat claims otherwise as unverified.145146When it is still worth doing: developer documentation, where users paste docs into coding assistants or agent tools fetch them directly. It is cheap to generate from your docs index.147148```markdown149# Example Docs150151> Example is an open-source job queue for Node.js.152153## Docs154155- [Quickstart](https://example.com/docs/quickstart.md): install and run a first job156- [API reference](https://example.com/docs/api.md): every public method157158## Optional159160- [Changelog](https://example.com/changelog.md)161```162163Generate it from the same source as the sitemap so it never drifts.164165### Step 7: Measure honestly166167- Server logs: count requests by user agent and status code. Verify identity where vendors publish IP ranges; user-agent strings can be spoofed.168- Referral traffic from AI products appears in analytics only when the product passes a referrer or UTM; absence of referrals is not proof of absence.169- Do not report "AI visibility scores" from third-party tools as fact; describe their method.170171## Best Practices1721731. Decide training, search, and user-fetch policy separately, in writing.1742. Name each AI agent explicitly; do not rely on the `*` group.1753. Re-check vendor crawler pages before each change; names and behaviors change.1764. Use authentication, not robots.txt, for content that must not be fetched.1775. Fix ordinary indexability first; AI search builds on it.1786. Keep factual claims dated and sourced so quotes stay accurate.1797. Treat llms.txt as optional convenience, not a visibility lever.180181## References182183Checked 2026-11-04 (crawler lists change often; re-verify):184185- OpenAI crawlers: https://developers.openai.com/api/docs/bots (formerly platform.openai.com/docs/bots)186- Anthropic crawlers: https://support.claude.com/en/articles/8896518187- Google common crawlers and Google-Extended: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers188- Google AI features and your website: https://developers.google.com/search/docs/appearance/ai-features189- Perplexity crawlers: https://docs.perplexity.ai/guides/bots190- Applebot: https://support.apple.com/en-us/119829191- Common Crawl CCBot: https://commoncrawl.org/ccbot192- llms.txt proposal: https://llmstxt.org/193- Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309194- Structured data guidelines: https://developers.google.com/search/docs/appearance/structured-data/sd-policies