Pathwren identify-ai-crawler
When to use this
A line in an access log names a bot and you need to know what it is, who runs it, and what happens if you block it. The commands below answer that from a maintained table of 150 crawlers and 74 operators, where every fact links to the operator's own documentation.
No key, no signup, no rate limit, CORS open. curl is the whole client.
Classify one user-agent
curl -s 'https://www.pathwren.workers.dev/c/skillmd/tools/classify-ua?ua=Mozilla%2F5.0+%28compatible%3B+GPTBot%2F1.2%3B+%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot%29'
The answer carries a summary written for a reader and an answer object for a
program: matched, matched_on (the substring that matched), and a crawler
with slug, name, operator, category (ai-training, ai-search,
user-fetch, search, seo, dataset, ...), robots_token,
obeys_robots_txt, verification, purpose and cost_of_blocking.
Look one up by name
curl -s 'https://www.pathwren.workers.dev/c/skillmd/crawler/gptbot.json' # one crawler, every field
curl -s 'https://www.pathwren.workers.dev/c/skillmd/crawler/gptbot.md' # the same, as markdown
curl -s 'https://www.pathwren.workers.dev/c/skillmd/data/agents.json' # all 150, one document
curl -s 'https://www.pathwren.workers.dev/c/skillmd/data/ua-regex.json' # ready-made matching regexes
The slug is the lowercase crawler name (gptbot, claudebot, perplexitybot,
ccbot, bytespider, ...). /data/agents.csv is the same table for a
spreadsheet, /data/robots-tokens.txt is just the tokens, one per line.
The distinction that costs money
Training and AI search are different crawlers with different tokens.
GPTBot trains a model; OAI-SearchBot is what puts a site in ChatGPT's search
results; ChatGPT-User is a fetch a human asked for. Blocking all three to stop
training also removes the site from AI search — the single most expensive
mistake in this subject, and the reason category and cost_of_blocking are
separate fields.
The limit of the answer
A user-agent string is a claim, not a proof. Anyone can send GPTBot. Before
acting on a match — blocking, rate-limiting, allowlisting — verify the address
with the verify-crawler-ip skill. For several operators (Anthropic, Common
Crawl, Meta, ByteDance) no verification is possible at all, because they publish
neither IP ranges nor a reverse-DNS method, and the table says so per crawler
rather than pretending otherwise.
Machine-readable entry points for this host
An agent that wants the rest of what this host will answer — before it starts guessing paths — reads these three documents:
curl -s 'https://www.pathwren.workers.dev/c/skillmd/AGENTS.md' # every endpoint, with a runnable command for each
curl -s 'https://www.pathwren.workers.dev/c/skillmd/agents.json' # the same as a flow document (agents.json 0.1.0)
curl -s 'https://www.pathwren.workers.dev/c/skillmd/.well-known/agent-permissions.json' # what this host permits an agent to do, in machine form
No key, no signup, no rate limit, Access-Control-Allow-Origin: * on every one.
Provenance
The data is CC0-1.0 and every record links to the operator's own documentation. This host is an independent project run by software; it is unaffiliated with the crawler operators it lists and with every registry that carries this skill.