AI Crawler Check

AI Crawler Check

Diagnose whether a website is accidentally blocking the crawlers that AI search products use to read and cite pages. Report findings per bot and give the exact robots.txt edit to fix any problem. This skill is diagnostic only. It never changes the user's site. It reads robots.txt and HTTP/meta directives, then presents findings and a suggested fix that the user applies themselves.

When to use

Use whenever someone wants to know if AI systems can access, read, index, or cite their website — especially campaign, small-business, or personal sites worried about AI-search visibility. Also use for "audit my robots.txt for AI crawlers" type requests.

What you need from the user

Just the website URL (e.g. example.com or https://example.com). If they haven't given one, ask for it. One domain per run.

How to run it

The bundled script does the fetching and evaluation. Run it against the URL:

python3 scripts/check_crawlers.py <URL>

Add --json if you want structured output to reformat yourself. The script:

1. Fetches <site>/robots.txt (a missing/404 robots.txt means everything is allowed).
2. Fetches the homepage to inspect the X-Robots-Tag response header and any <meta name="robots"> tags.
3. Evaluates every known AI crawler user-agent against the robots.txt rules using the standard longest-match algorithm (so Allow: can override a broader Disallow:, and an explicit per-bot group overrides the * group).
4. Prints per-bot Allowed/Blocked status, flags indexing-suppressing directives, and prints a suggested robots.txt edit for any blocked search/retrieval bots.

Environment requirements

This skill assumes the standard skills runtime: a filesystem (so the bundled scripts/ and bots.json are readable) and outbound network access (so the script can fetch live robots.txt and homepage headers). Both are available in the environment this skill is designed for, and the fetch is confirmed working.

If you are running somewhere without outbound network access, do not try to fetch — go straight to the paste fallback below. If you are running somewhere without a filesystem to execute the script, reproduce its logic manually using the bot table in scripts/bots.json and the longest-match rule described above; do not skip the evaluation.

If the live fetch fails — never fabricate

Some sites (e.g. those behind a WAF or bot filter) return 403/timeout to automated fetchers even though a browser can reach them. A certificate-verify error (common on local Python installs that haven't run their SSL cert setup step, e.g. macOS python.org builds without Install Certificates.command) is the same kind of failure — it means the environment couldn't fetch, not that the site actually blocks anything. If the script cannot fetch robots.txt, it exits with an error and does not invent one — and neither should you. Never simulate, guess at, or hallucinate the contents of a robots.txt or the headers. Instead:

1. Tell the user the live fetch failed and why.
2. Ask them to open <site>/robots.txt in their browser and paste the contents (or upload the file).
3. Save what they paste to a file and re-run: python3 scripts/check_crawlers.py <URL> --robots-file <path>.

When evaluating a pasted file, the report clearly marks that robots.txt was user-supplied rather than fetched live, and notes that the header/meta scan may not have run. Pass those caveats along — an unknown directive is "unknown," not "clear."

How to present the results

Lead with the headline: is anything that matters for AI visibility blocked? Then walk through the three groups. Keep the training-vs-retrieval distinction front and center — it's the whole point of the skill.

Follow this structure so every run reads consistently. Fill the bracketed parts from the script output; keep the per-bot consequence lines — do not flatten the report into a bare status grid, because the reason a blocked search bot matters more than a blocked training bot is the value of the skill.

## AI Crawler Check — <site>

**Verdict:** <one line: e.g. "2 AI search crawlers are blocked — ChatGPT and
Perplexity can't cite your site" OR "All clear — no AI search crawlers blocked.">

**robots.txt:** <found / not found (all allowed by default) / user-supplied>

### AI search & retrieval — you almost certainly want these ALLOWED
- <✅/🚫> **<Bot>** (<vendor>) — <consequence line if blocked>
  (OAI-SearchBot, Claude-SearchBot, PerplexityBot)

### User-triggered fetch — allow so AI can open your page on request
- <✅/🚫> **<Bot>** (<vendor>) — <consequence + robots.txt caveat if relevant>
  (ChatGPT-User, Claude-User, Perplexity-User)

### AI training — your call; blocking these is a legitimate choice
- <✅/🚫> **<Bot>** (<vendor>) — blocking = out of training data, no effect on citations
  (GPTBot, ClaudeBot, Google-Extended)

### Indexing directives (headers / meta)
<Report X-Robots-Tag / meta-robots findings ONLY if the scan ran. If it was
skipped because of a WAF/challenge page or an unreachable homepage, say so and
mark these UNKNOWN — never report a challenge page's noindex as the site's.>

### Suggested fix (you apply this — nothing was changed)
<Only if search/user bots are blocked. Show the robots.txt snippet verbatim.>

Scope note: this check evaluates access to the site root (/) only. A robots.txt can allow the root while disallowing specific sections (e.g. /press/, /issues/, /blog/) — a common pattern on CMS-driven sites. Add one line to every report: "This checks root-level access only — if you suspect a specific section (like your press or issues pages) is blocked, share that path and re-run the check against it."

Notes on the groups:

• AI search / retrieval bots decide whether the site can appear in and be cited by ChatGPT Search, Claude search, and Perplexity answers. If one is blocked, say plainly what it costs — e.g. "Your site can't appear in ChatGPT's search answers."
• User-triggered fetchers fetch a page live when a user asks the AI to open that specific URL. (Note: OpenAI and Perplexity say robots.txt may not fully govern their user-fetchers — mention this rather than overpromising.)
• AI training bots control whether content trains models. Blocking these is a legitimate choice and does NOT affect search citations. Don't tell the user to unblock them. Google-Extended in particular does not affect normal Google Search ranking.

The script only ever suggests unblocking search and user-fetch bots — never training bots — so a candidate makes an informed choice instead of blindly unblocking everything. Remind the user they apply the edit; the tool changed nothing. If a User-agent: group for that exact bot already exists in the file, tell the user to edit that existing group directly instead of appending a duplicate — most parsers resolve duplicates correctly, but not all do, so don't rely on it.

Worked example

Blocked site — campaign-example.org has Disallow: / for GPTBot and OAI-SearchBot in its robots.txt (a WordPress security plugin added these by default):

## AI Crawler Check — campaign-example.org

**Verdict:** 1 AI search crawler is blocked — ChatGPT can't cite your site in search answers.

**robots.txt:** found

### AI search & retrieval — you almost certainly want these ALLOWED
- 🚫 **OAI-SearchBot** (OpenAI) — Your site will NOT appear in ChatGPT search answers.
- ✅ **Claude-SearchBot** (Anthropic)
- ✅ **PerplexityBot** (Perplexity)

### User-triggered fetch — allow so AI can open your page on request
- ✅ **ChatGPT-User** (OpenAI)
- ✅ **Claude-User** (Anthropic)
- ✅ **Perplexity-User** (Perplexity)

### AI training — your call; blocking these is a legitimate choice
- 🚫 **GPTBot** (OpenAI) — Your content won't be used to train OpenAI models. No effect on ChatGPT search citations.
- ✅ **ClaudeBot** (Anthropic)
- ✅ **Google-Extended** (Google)

### Indexing directives (headers / meta)
No indexing-suppressing header or meta flags found.

### Suggested fix (you apply this — nothing was changed)

Add or update these blocks in your robots.txt to restore AI search visibility.

User-agent: OAI-SearchBot Allow: /

**Clean site** — `example-clean.org` blocks nothing:

AI Crawler Check — example-clean.org

Verdict: All clear — no AI search crawlers blocked.

robots.txt: not found (all allowed by default)

[... all three groups show ✅ ...]

No AI search/retrieval crawlers are blocked in robots.txt. ✓ (No indexing-suppressing header/meta flags found either.)

### Scope of header / meta findings — don't overclaim

`robots.txt` applies site-wide. An `X-Robots-Tag` header applies to the response
it was served on, and a `<meta name="robots">` tag applies **only to the page it
sits on** — here, the homepage. So a homepage `noindex` does **not** prove the
whole site is blocked. It's a strong signal (often a CMS-wide default worth
investigating), but state it as "found on your homepage" rather than "your entire
site is set to noindex." The script's output already scopes this language — keep
it.

If the homepage fetch hit a WAF/CDN challenge page (Cloudflare, Sucuri, etc.),
the script **skips** the header/meta scan on purpose: those interstitials carry
their own `noindex` for the captcha screen, and reporting it would be a false
positive. When the scan is skipped, tell the user the directives are UNKNOWN and
suggest they view the homepage source in a browser — don't guess.

## Accuracy rules (important)

- Be precise about which user-agent does what. The reference data lives in
  `scripts/bots.json`; trust it over memory. The key distinction: training bots
  vs. search/retrieval bots vs. user-fetchers.
- If asked about a bot not in the list, or if a bot's role is genuinely
  ambiguous, **say so** rather than guessing. `bots.json` carries a `confidence`
  field and inline caveats (e.g. sources disagree on whether PerplexityBot is a
  pure indexer or retrieval crawler) — pass those caveats along honestly.
- The AI-crawler landscape changes. Tokens and roles here are current as of
  mid-2026. If the user needs certainty on a newly launched or renamed bot, web
  search the vendor's official crawler documentation to confirm before asserting.
- Never claim you changed or fixed the site. You produce a diagnosis and a
  suggested edit; the user applies it.

## Reference

- `scripts/check_crawlers.py` — the diagnostic (fetch, parse, evaluate, report).
  Flags: `--json` (structured output); `--robots-file <path>` (evaluate a
  user-supplied robots.txt instead of fetching live, for when a site blocks
  automated fetches).
- `scripts/bots.json` — the authoritative per-bot data: token, vendor, category
  (training/search/user_fetch), plain-language description, cost of blocking,
  and a confidence flag. Edit this file to add or update bots as the landscape
  shifts.

## Runtime & scope

- **Posture:** Diagnostic only. It reads a site's `robots.txt` and headers and reports findings plus a suggested fix; it never modifies the user's site.
- **Neutrality:** Politically neutral — it evaluates crawler *access*, never content, positions, or claims.
- **Runtime:** Python 3.6+, standard library only. Optionally uses `certifi` as an HTTPS-certificate fallback if it happens to be installed; not required.