Pathwren robots-txt-for-ai
When to use this
Someone needs a robots.txt that stops AI training without disappearing from AI search, or an existing file needs review. The tokens change often enough that a hand-written block goes stale; these are generated from the same table of 150 crawlers that the rest of this host publishes and rebuilt every six hours.
Take a ready-made policy
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/block-ai-training.txt' # no training; AI search still allowed
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/allow-ai-search-only.txt' # search and assistants in, datasets out
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/block-all-ai.txt' # every AI token, blocked
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/maximum-ai-visibility.txt' # allow everything AI
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/block-datasets.txt' # dataset crawlers only
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/block-seo-tools.txt' # SEO crawlers only
curl -s 'https://www.pathwren.workers.dev/c/skillmd/robots/allow-all.txt' # the empty policy, for comparison
Paste the file in as-is. https://www.pathwren.workers.dev/c/skillmd/policy/block-ai-training.md explains what
each stance includes and what it costs, one page per stance.
Lint a file that already exists
curl -s --get 'https://www.pathwren.workers.dev/c/skillmd/tools/robots-lint' --data-urlencode "robots_txt=$(cat robots.txt)"
Reports the rules that do not do what their author expected: unknown or
misspelled tokens, directives no crawler implements (Noindex:, Crawl-delay
where it is ignored), a group that shadows another, and AI tokens the file never
mentions at all.
Ask whether one crawler may fetch one path
curl -s --get 'https://www.pathwren.workers.dev/c/skillmd/tools/robots-allowed' --data-urlencode "robots_txt=$(cat robots.txt)" --data-urlencode 'ua=GPTBot' --data-urlencode 'path=/blog/post'
And the whole-table version — every AI crawler against one path at once, which is the review that catches "we blocked ourselves out of AI search":
curl -s --get 'https://www.pathwren.workers.dev/c/skillmd/tools/ai-access' --data-urlencode "robots_txt=$(cat robots.txt)" --data-urlencode 'path=/'
The limit of the answer
robots.txt is a request, not a control. Perplexity-User and Bytespider are
documented or reported as not governed by it, so a rule for them is a statement
of intent and enforcement has to happen at the edge — https://www.pathwren.workers.dev/c/skillmd/snippet/ has the
nginx, Caddy, Apache and Cloudflare Worker versions. Nothing here modifies your
file: these endpoints read what you send and answer; the writing is yours.
Machine-readable entry points for this host
An agent that wants the rest of what this host will answer — before it starts guessing paths — reads these three documents:
curl -s 'https://www.pathwren.workers.dev/c/skillmd/AGENTS.md' # every endpoint, with a runnable command for each
curl -s 'https://www.pathwren.workers.dev/c/skillmd/agents.json' # the same as a flow document (agents.json 0.1.0)
curl -s 'https://www.pathwren.workers.dev/c/skillmd/.well-known/agent-permissions.json' # what this host permits an agent to do, in machine form
No key, no signup, no rate limit, Access-Control-Allow-Origin: * on every one.
Provenance
The data is CC0-1.0 and every record links to the operator's own documentation. This host is an independent project run by software; it is unaffiliated with the crawler operators it lists and with every registry that carries this skill.