# AI Search Visibility

> Decide and implement how a site is accessed by AI crawlers, AI search features, and answer engines: which user agents to allow or block in robots.txt (training vs search vs user-triggered fetchers), what Google's AI features actually require, an honest take on llms.txt, and content/structured-data practices that make pages easy to quote accurately. Use this skill when someone asks how to appear in ChatGPT search, Claude, Perplexity, Google AI Overviews or AI Mode, wants to block AI training without losing search visibility, or asks whether they need llms.txt. Activate when: AI search, answer engine optimization, AEO, GEO, generative engine optimization, AI Overviews, AI Mode, GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, llms.txt, block AI crawlers.

- Skill: `latestaiagents/ai-search-visibility` (Agent Skill)
- Install (CLI): `npx get-skill add latestaiagents/ai-search-visibility`
- Raw SKILL.md: https://api.skillmd.com/api/skills/latestaiagents/ai-search-visibility/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- Author: latestaiagents (https://skillmd.com/u/latestaiagents)
- Updated: 2026-11-14
- Page: https://skillmd.com/skills/latestaiagents/ai-search-visibility

---


# AI Search Visibility

**Separate "may my content train models?" from "may AI search tools find and cite me?", then configure each crawler on purpose.**

There is no verified ranking formula for AI answers. This skill covers the parts you control and can verify: crawler access, indexability, and clear, quotable content.

## When to Use

- Choosing a robots.txt policy for AI crawlers
- "We blocked AI bots and disappeared from ChatGPT/Perplexity search" (or the reverse)
- Evaluating whether to publish `llms.txt`
- Making docs or product pages easier for answer engines to quote correctly

## When NOT to Use

- General crawl/index problems in Google Search (use `technical-seo-audit`)
- Legal questions about copyright or licensing; this skill describes vendor-documented controls only

## Workflow

### Step 1: Decide the policy per purpose

Most vendors now separate crawlers by purpose. Agree on three decisions with the site owner:

| Purpose | Question |
|---------|----------|
| Model training | May future models be trained on this content? |
| AI search / indexing | May AI search products index and link to this content? |
| User-triggered fetch | When a user asks an assistant to read a URL, may it fetch the page? |

Blocking training does not have to mean blocking search, and vice versa.

### Step 2: Map purposes to documented user agents

As checked on 2026-11-04 from each vendor's official page. Vendors add and rename agents; re-verify before shipping.

| Vendor | Token | Purpose (per vendor docs) | robots.txt |
|--------|-------|---------------------------|------------|
| OpenAI | `GPTBot` | Content that may be used to train generative AI foundation models | Honored; disallow signals "do not use for training" |
| OpenAI | `OAI-SearchBot` | Surfacing sites in ChatGPT search features | Honored; disallow removes you from ChatGPT search answers |
| OpenAI | `ChatGPT-User` | User actions in ChatGPT / custom GPTs | OpenAI states robots.txt rules may not apply to user-initiated requests |
| OpenAI | `OAI-AdsBot` | Checking ad landing pages; OpenAI says data is not used for training | See OpenAI docs |
| Anthropic | `ClaudeBot` | Collecting content that may contribute to model training | Disallow excludes future site content from training datasets |
| Anthropic | `Claude-SearchBot` | Indexing to improve search result quality | Disallow prevents indexing for search |
| Anthropic | `Claude-User` | Fetching pages when a Claude user asks | Disallow prevents retrieval in response to user queries |
| Google | `Google-Extended` | Product token: Gemini training and grounding in Gemini Apps / Vertex AI | Does not affect Google Search inclusion or ranking; not a separate crawler |
| Perplexity | `PerplexityBot` | Surfacing and linking sites in Perplexity search results | Honored |
| Perplexity | `Perplexity-User` | Fetching pages for user requests | Perplexity states it generally ignores robots.txt |
| Apple | `Applebot` | Spotlight, Siri, Safari search features | Honored |
| Apple | `Applebot-Extended` | Opt out of training Apple's foundation models | Pages can still appear in Apple search results |
| Common Crawl | `CCBot` | Open web crawl used by many downstream datasets | Honored via `Disallow` |

Key consequences:

- Google AI Overviews and AI Mode are part of Google Search. `Google-Extended` does not control them. To limit how Search content appears there, Google points to `nosnippet`, `data-nosnippet`, `max-snippet`, or `noindex`, which also affect regular Search snippets.
- User-triggered fetchers may not obey robots.txt (OpenAI and Perplexity say so explicitly). Use authentication if content must not be fetched.
- Anthropic notes that IP blocking may not work as a reliable opt-out because it also blocks reading robots.txt.

### Step 3: Write the robots.txt

A crawler follows only the most specific group that matches it, so give each AI agent its own group rather than relying on `User-agent: *`.

Example: allow AI search and user fetches, opt out of training.

```text
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/

User-agent: *
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml
```

Notes:

- Multiple `User-agent` lines may share one group (RFC 9309).
- `Allow: /` plus a narrower `Disallow` works because the longest matching rule wins.
- `Crawl-delay` is non-standard. Anthropic documents support for it; Google does not support it. Others: check their docs.
- Disallowing `CCBot` affects future Common Crawl snapshots, not copies already published.

Verify with the live file:

```bash
curl -s https://www.example.com/robots.txt
```

### Step 4: Make sure search-oriented AI can actually use the page

AI search features generally build on ordinary crawling and indexing. For Google specifically, a page must be indexed and eligible for a snippet to be a supporting link in AI Overviews or AI Mode, and Google states there are no extra requirements, no special schema, and no "AI text files" needed.

So the checklist is the normal one:

- 200 status, indexable, not blocked, canonical set correctly
- Main content in server-rendered HTML (not every crawler runs JavaScript)
- Accurate `<title>`, meta description, and headings

### Step 5: Write content that is easy to quote correctly

These are editorial practices, not ranking guarantees:

- State the answer plainly near the top, then the detail. One claim per sentence where accuracy matters (prices, limits, versions, dates).
- Put facts in text, not only in images or interactive widgets.
- Date-stamp facts that change ("as of 2026-11-04") and keep them current.
- Use descriptive headings that match how people phrase questions.
- Cite primary sources for claims.
- Add JSON-LD that matches visible content (`Organization`, `Product`, `Article`, `SoftwareApplication`, `BreadcrumbList`). Structured data helps machines parse the page; it does not guarantee inclusion anywhere.

### Step 6: llms.txt (optional, low priority)

Facts:

- `llms.txt` is a proposal by Jeremy Howard (published September 2024 at llmstxt.org), not an IETF or W3C standard.
- Format: a markdown file at `/llms.txt` with a required H1 (site name), an optional blockquote summary, and optional H2 sections listing links. The proposal also suggests serving markdown versions of pages (`page.md` or `page.html.md`).
- Google's AI features documentation says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode.
- As of 2026-11-04 we found no documentation from Google, OpenAI, Anthropic, Perplexity, or Apple stating that their search crawlers use `llms.txt` for ranking or inclusion. Treat claims otherwise as unverified.

When it is still worth doing: developer documentation, where users paste docs into coding assistants or agent tools fetch them directly. It is cheap to generate from your docs index.

```markdown
# Example Docs

> Example is an open-source job queue for Node.js.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): install and run a first job
- [API reference](https://example.com/docs/api.md): every public method

## Optional

- [Changelog](https://example.com/changelog.md)
```

Generate it from the same source as the sitemap so it never drifts.

### Step 7: Measure honestly

- Server logs: count requests by user agent and status code. Verify identity where vendors publish IP ranges; user-agent strings can be spoofed.
- Referral traffic from AI products appears in analytics only when the product passes a referrer or UTM; absence of referrals is not proof of absence.
- Do not report "AI visibility scores" from third-party tools as fact; describe their method.

## Best Practices

1. Decide training, search, and user-fetch policy separately, in writing.
2. Name each AI agent explicitly; do not rely on the `*` group.
3. Re-check vendor crawler pages before each change; names and behaviors change.
4. Use authentication, not robots.txt, for content that must not be fetched.
5. Fix ordinary indexability first; AI search builds on it.
6. Keep factual claims dated and sourced so quotes stay accurate.
7. Treat llms.txt as optional convenience, not a visibility lever.

## References

Checked 2026-11-04 (crawler lists change often; re-verify):

- OpenAI crawlers: https://developers.openai.com/api/docs/bots (formerly platform.openai.com/docs/bots)
- Anthropic crawlers: https://support.claude.com/en/articles/8896518
- Google common crawlers and Google-Extended: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- Google AI features and your website: https://developers.google.com/search/docs/appearance/ai-features
- Perplexity crawlers: https://docs.perplexity.ai/guides/bots
- Applebot: https://support.apple.com/en-us/119829
- Common Crawl CCBot: https://commoncrawl.org/ccbot
- llms.txt proposal: https://llmstxt.org/
- Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
- Structured data guidelines: https://developers.google.com/search/docs/appearance/structured-data/sd-policies

