# Robots Txt For AI

> Write or audit the AI section of a robots.txt: start from a ready-made policy (block AI training but keep AI search, block everything, maximum AI visibility), lint an existing file for rules that do not do what their author thinks, and check whether a named crawler may fetch a given path. Use when editing robots.txt for AI bots or reviewing one in a pull request.

- Skill: `pathwren/robots-txt-for-ai` (Agent Skill)
- Install (CLI): `npx skillmds add pathwren/robots-txt-for-ai`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pathwren/robots-txt-for-ai/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: any agent that reads SKILL.md (Claude Code, Claude.ai, Cursor, Codex, Windsurf, 60+ more)
- Category: Docs & Writing
- License: CC0-1.0
- Author: Pathwren (https://skillmd.com/u/pathwren)
- Updated: 2026-09-03
- Page: https://skillmd.com/skills/pathwren/robots-txt-for-ai

---


# robots-txt-for-ai

## When to use this

Someone needs a robots.txt that stops AI training without disappearing from AI
search, or an existing file needs review. The tokens change often enough that a
hand-written block goes stale; these are generated from the same table of
150 crawlers that the rest of this host publishes and rebuilt every six
hours.

## Take a ready-made policy

```bash
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/block-ai-training.txt'      # no training; AI search still allowed
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/allow-ai-search-only.txt'   # search and assistants in, datasets out
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/block-all-ai.txt'           # every AI token, blocked
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/maximum-ai-visibility.txt'  # allow everything AI
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/block-datasets.txt'         # dataset crawlers only
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/block-seo-tools.txt'        # SEO crawlers only
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/allow-all.txt'              # the empty policy, for comparison
```

Paste the file in as-is. `https://www.pathwren.workers.dev/c/skillmd/policy/block-ai-training.md` explains what
each stance includes and what it costs, one page per stance.

## Lint a file that already exists

```bash
curl -s --get 'https://www.pathwren.workers.dev/c/skillmd/tools/robots-lint' --data-urlencode "robots_txt=$(cat robots.txt)"
```

Reports the rules that do not do what their author expected: unknown or
misspelled tokens, directives no crawler implements (`Noindex:`, `Crawl-delay`
where it is ignored), a group that shadows another, and AI tokens the file never
mentions at all.

## Ask whether one crawler may fetch one path

```bash
curl -s --get 'https://www.pathwren.workers.dev/c/skillmd/tools/robots-allowed' --data-urlencode "robots_txt=$(cat robots.txt)" --data-urlencode 'ua=GPTBot' --data-urlencode 'path=/blog/post'
```

And the whole-table version — every AI crawler against one path at once, which is
the review that catches "we blocked ourselves out of AI search":

```bash
curl -s --get 'https://www.pathwren.workers.dev/c/skillmd/tools/ai-access' --data-urlencode "robots_txt=$(cat robots.txt)" --data-urlencode 'path=/'
```

## The limit of the answer

robots.txt is a request, not a control. `Perplexity-User` and `Bytespider` are
documented or reported as not governed by it, so a rule for them is a statement
of intent and enforcement has to happen at the edge — `https://www.pathwren.workers.dev/c/skillmd/snippet/` has the
nginx, Caddy, Apache and Cloudflare Worker versions. Nothing here modifies your
file: these endpoints read what you send and answer; the writing is yours.

## Machine-readable entry points for this host

An agent that wants the rest of what this host will answer — before it starts
guessing paths — reads these three documents:

```bash
curl -s 'https://www.pathwren.workers.dev/c/skillmd/AGENTS.md'                          # every endpoint, with a runnable command for each
curl -s 'https://www.pathwren.workers.dev/c/skillmd/agents.json'                        # the same as a flow document (agents.json 0.1.0)
curl -s 'https://www.pathwren.workers.dev/c/skillmd/.well-known/agent-permissions.json' # what this host permits an agent to do, in machine form
```

No key, no signup, no rate limit, `Access-Control-Allow-Origin: *` on every one.

## Provenance

The data is CC0-1.0 and every record links to the operator's own documentation.
This host is an independent project run by software; it is unaffiliated with the
crawler operators it lists and with every registry that carries this skill.