# AI Crawler Check > Diagnose whether a website is accidentally blocking AI search crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended). Use this whenever a user asks if AI can read/find/cite their site, whether ChatGPT or Claude or Perplexity can see their pages, why their site isn't showing up in AI search or AI answers, or asks to check/audit their robots.txt for AI or LLM crawler access. Trigger it for phrases like "can ChatGPT read my site", "is my site blocking AI", "check my robots.txt for AI crawlers", "why don't I show up in AI search", or when someone shares a campaign/business URL and worries about AI visibility. Diagnostic only — it inspects robots.txt and headers and reports findings plus a suggested fix; it never modifies the user's site. - Skill: `meturley/ai-crawler-check` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add meturley/ai-crawler-check` - Raw SKILL.md: https://api.skillmd.com/api/skills/meturley/ai-crawler-check/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Marketing & Growth - Author: meturley (https://skillmd.com/u/meturley) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/meturley/ai-crawler-check --- # AI Crawler Check Diagnose whether a website is accidentally blocking the crawlers that AI search products use to read and cite pages. Report findings per bot and give the exact `robots.txt` edit to fix any problem. **This skill is diagnostic only. It never changes the user's site.** It reads `robots.txt` and HTTP/meta directives, then presents findings and a suggested fix that the user applies themselves. ## When to use Use whenever someone wants to know if AI systems can access, read, index, or cite their website — especially campaign, small-business, or personal sites worried about AI-search visibility. Also use for "audit my robots.txt for AI crawlers" type requests. ## What you need from the user Just the website URL (e.g. `example.com` or `https://example.com`). If they haven't given one, ask for it. One domain per run. ## How to run it The bundled script does the fetching and evaluation. Run it against the URL: ```bash python3 scripts/check_crawlers.py ``` Add `--json` if you want structured output to reformat yourself. The script: 1. Fetches `/robots.txt` (a missing/404 robots.txt means everything is allowed). 2. Fetches the homepage to inspect the `X-Robots-Tag` response header and any `` tags. 3. Evaluates every known AI crawler user-agent against the robots.txt rules using the standard longest-match algorithm (so `Allow:` can override a broader `Disallow:`, and an explicit per-bot group overrides the `*` group). 4. Prints per-bot Allowed/Blocked status, flags indexing-suppressing directives, and prints a suggested robots.txt edit for any blocked **search/retrieval** bots. ### Environment requirements This skill assumes the standard skills runtime: a filesystem (so the bundled `scripts/` and `bots.json` are readable) and outbound network access (so the script can fetch live `robots.txt` and homepage headers). Both are available in the environment this skill is designed for, and the fetch is confirmed working. If you are running somewhere **without** outbound network access, do not try to fetch — go straight to the paste fallback below. If you are running somewhere without a filesystem to execute the script, reproduce its logic manually using the bot table in `scripts/bots.json` and the longest-match rule described above; do not skip the evaluation. ### If the live fetch fails — never fabricate Some sites (e.g. those behind a WAF or bot filter) return 403/timeout to automated fetchers even though a browser can reach them. A certificate-verify error (common on local Python installs that haven't run their SSL cert setup step, e.g. macOS python.org builds without `Install Certificates.command`) is the same kind of failure — it means the environment couldn't fetch, not that the site actually blocks anything. If the script cannot fetch `robots.txt`, it exits with an error and does **not** invent one — and neither should you. **Never simulate, guess at, or hallucinate the contents of a robots.txt or the headers.** Instead: 1. Tell the user the live fetch failed and why. 2. Ask them to open `/robots.txt` in their browser and paste the contents (or upload the file). 3. Save what they paste to a file and re-run: `python3 scripts/check_crawlers.py --robots-file `. When evaluating a pasted file, the report clearly marks that robots.txt was user-supplied rather than fetched live, and notes that the header/meta scan may not have run. Pass those caveats along — an unknown directive is "unknown," not "clear." ## How to present the results Lead with the headline: is anything that matters for AI visibility blocked? Then walk through the three groups. **Keep the training-vs-retrieval distinction front and center — it's the whole point of the skill.** Follow this structure so every run reads consistently. Fill the bracketed parts from the script output; keep the per-bot consequence lines — do not flatten the report into a bare status grid, because the *reason* a blocked search bot matters more than a blocked training bot is the value of the skill. ``` ## AI Crawler Check — **Verdict:** **robots.txt:** ### AI search & retrieval — you almost certainly want these ALLOWED - <✅/🚫> **** () — (OAI-SearchBot, Claude-SearchBot, PerplexityBot) ### User-triggered fetch — allow so AI can open your page on request - <✅/🚫> **** () — (ChatGPT-User, Claude-User, Perplexity-User) ### AI training — your call; blocking these is a legitimate choice - <✅/🚫> **** () — blocking = out of training data, no effect on citations (GPTBot, ClaudeBot, Google-Extended) ### Indexing directives (headers / meta) ### Suggested fix (you apply this — nothing was changed) ``` **Scope note:** this check evaluates access to the site root (`/`) only. A robots.txt can allow the root while disallowing specific sections (e.g. `/press/`, `/issues/`, `/blog/`) — a common pattern on CMS-driven sites. Add one line to every report: "This checks root-level access only — if you suspect a specific section (like your press or issues pages) is blocked, share that path and re-run the check against it." Notes on the groups: - **AI search / retrieval bots** decide whether the site can appear in and be cited by ChatGPT Search, Claude search, and Perplexity answers. If one is blocked, say plainly what it costs — e.g. "Your site can't appear in ChatGPT's search answers." - **User-triggered fetchers** fetch a page live when a user asks the AI to open that specific URL. (Note: OpenAI and Perplexity say robots.txt may not fully govern their user-fetchers — mention this rather than overpromising.) - **AI training bots** control whether content trains models. Blocking these is a **legitimate choice** and does NOT affect search citations. Don't tell the user to unblock them. `Google-Extended` in particular does not affect normal Google Search ranking. The script only ever suggests unblocking search and user-fetch bots — never training bots — so a candidate makes an informed choice instead of blindly unblocking everything. Remind the user **they** apply the edit; the tool changed nothing. If a `User-agent:` group for that exact bot already exists in the file, tell the user to edit that existing group directly instead of appending a duplicate — most parsers resolve duplicates correctly, but not all do, so don't rely on it. ### Worked example **Blocked site** — `campaign-example.org` has `Disallow: /` for `GPTBot` and `OAI-SearchBot` in its robots.txt (a WordPress security plugin added these by default): ``` ## AI Crawler Check — campaign-example.org **Verdict:** 1 AI search crawler is blocked — ChatGPT can't cite your site in search answers. **robots.txt:** found ### AI search & retrieval — you almost certainly want these ALLOWED - 🚫 **OAI-SearchBot** (OpenAI) — Your site will NOT appear in ChatGPT search answers. - ✅ **Claude-SearchBot** (Anthropic) - ✅ **PerplexityBot** (Perplexity) ### User-triggered fetch — allow so AI can open your page on request - ✅ **ChatGPT-User** (OpenAI) - ✅ **Claude-User** (Anthropic) - ✅ **Perplexity-User** (Perplexity) ### AI training — your call; blocking these is a legitimate choice - 🚫 **GPTBot** (OpenAI) — Your content won't be used to train OpenAI models. No effect on ChatGPT search citations. - ✅ **ClaudeBot** (Anthropic) - ✅ **Google-Extended** (Google) ### Indexing directives (headers / meta) No indexing-suppressing header or meta flags found. ### Suggested fix (you apply this — nothing was changed) ``` # Add or update these blocks in your robots.txt to restore AI search visibility. User-agent: OAI-SearchBot Allow: / ``` **Clean site** — `example-clean.org` blocks nothing: ``` ## AI Crawler Check — example-clean.org **Verdict:** All clear — no AI search crawlers blocked. **robots.txt:** not found (all allowed by default) [... all three groups show ✅ ...] No AI search/retrieval crawlers are blocked in robots.txt. ✓ (No indexing-suppressing header/meta flags found either.) ``` ### Scope of header / meta findings — don't overclaim `robots.txt` applies site-wide. An `X-Robots-Tag` header applies to the response it was served on, and a `` tag applies **only to the page it sits on** — here, the homepage. So a homepage `noindex` does **not** prove the whole site is blocked. It's a strong signal (often a CMS-wide default worth investigating), but state it as "found on your homepage" rather than "your entire site is set to noindex." The script's output already scopes this language — keep it. If the homepage fetch hit a WAF/CDN challenge page (Cloudflare, Sucuri, etc.), the script **skips** the header/meta scan on purpose: those interstitials carry their own `noindex` for the captcha screen, and reporting it would be a false positive. When the scan is skipped, tell the user the directives are UNKNOWN and suggest they view the homepage source in a browser — don't guess. ## Accuracy rules (important) - Be precise about which user-agent does what. The reference data lives in `scripts/bots.json`; trust it over memory. The key distinction: training bots vs. search/retrieval bots vs. user-fetchers. - If asked about a bot not in the list, or if a bot's role is genuinely ambiguous, **say so** rather than guessing. `bots.json` carries a `confidence` field and inline caveats (e.g. sources disagree on whether PerplexityBot is a pure indexer or retrieval crawler) — pass those caveats along honestly. - The AI-crawler landscape changes. Tokens and roles here are current as of mid-2026. If the user needs certainty on a newly launched or renamed bot, web search the vendor's official crawler documentation to confirm before asserting. - Never claim you changed or fixed the site. You produce a diagnosis and a suggested edit; the user applies it. ## Reference - `scripts/check_crawlers.py` — the diagnostic (fetch, parse, evaluate, report). Flags: `--json` (structured output); `--robots-file ` (evaluate a user-supplied robots.txt instead of fetching live, for when a site blocks automated fetches). - `scripts/bots.json` — the authoritative per-bot data: token, vendor, category (training/search/user_fetch), plain-language description, cost of blocking, and a confidence flag. Edit this file to add or update bots as the landscape shifts. ## Runtime & scope - **Posture:** Diagnostic only. It reads a site's `robots.txt` and headers and reports findings plus a suggested fix; it never modifies the user's site. - **Neutrality:** Politically neutral — it evaluates crawler *access*, never content, positions, or claims. - **Runtime:** Python 3.6+, standard library only. Optionally uses `certifi` as an HTTPS-certificate fallback if it happens to be installed; not required.