# Identify AI Crawler

> Identify the AI crawler, agent or bot behind a user-agent string or an access-log line: which operator runs it, what it is for, whether it obeys robots.txt, which robots.txt token blocks it, and what blocking it actually costs. Use when triaging bot traffic, naming an unknown client in a log, or deciding whether a hit was a training crawler or an AI search fetch — a distinction that costs people money.

- Skill: `a-60df050793770d0c51011064c702327f/identify-ai-crawler` (Agent Skill)
- Install (CLI): `npx skillmds add a-60df050793770d0c51011064c702327f/identify-ai-crawler`
- Raw SKILL.md: https://api.skillmd.com/api/skills/a-60df050793770d0c51011064c702327f/identify-ai-crawler/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: any agent that reads SKILL.md (Claude Code, Claude.ai, Cursor, Codex, Windsurf, 60+ more)
- Category: Data & Analytics
- License: CC0-1.0
- Author: Pathwren (https://skillmd.com/u/pathwren)
- Updated: 2026-09-03
- Page: https://skillmd.com/skills/a-60df050793770d0c51011064c702327f/identify-ai-crawler

---


# identify-ai-crawler

## When to use this

A line in an access log names a bot and you need to know what it is, who runs
it, and what happens if you block it. The commands below answer that from a
maintained table of 150 crawlers and 74 operators, where every fact
links to the operator's own documentation.

No key, no signup, no rate limit, CORS open. `curl` is the whole client.

## Classify one user-agent

```bash
curl -s 'https://www.pathwren.workers.dev/c/skillmd/tools/classify-ua?ua=Mozilla%2F5.0+%28compatible%3B+GPTBot%2F1.2%3B+%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot%29'
```

The answer carries a `summary` written for a reader and an `answer` object for a
program: `matched`, `matched_on` (the substring that matched), and a `crawler`
with `slug`, `name`, `operator`, `category` (`ai-training`, `ai-search`,
`user-fetch`, `search`, `seo`, `dataset`, ...), `robots_token`,
`obeys_robots_txt`, `verification`, `purpose` and `cost_of_blocking`.

## Look one up by name

```bash
curl -s 'https://www.pathwren.workers.dev/c/skillmd/crawler/gptbot.json'      # one crawler, every field
curl -s 'https://www.pathwren.workers.dev/c/skillmd/crawler/gptbot.md'        # the same, as markdown
curl -s 'https://www.pathwren.workers.dev/c/skillmd/data/agents.json'         # all 150, one document
curl -s 'https://www.pathwren.workers.dev/c/skillmd/data/ua-regex.json'       # ready-made matching regexes
```

The slug is the lowercase crawler name (`gptbot`, `claudebot`, `perplexitybot`,
`ccbot`, `bytespider`, ...). `/data/agents.csv` is the same table for a
spreadsheet, `/data/robots-tokens.txt` is just the tokens, one per line.

## The distinction that costs money

Training and AI search are **different crawlers with different tokens**.
`GPTBot` trains a model; `OAI-SearchBot` is what puts a site in ChatGPT's search
results; `ChatGPT-User` is a fetch a human asked for. Blocking all three to stop
training also removes the site from AI search — the single most expensive
mistake in this subject, and the reason `category` and `cost_of_blocking` are
separate fields.

## The limit of the answer

**A user-agent string is a claim, not a proof.** Anyone can send `GPTBot`. Before
acting on a match — blocking, rate-limiting, allowlisting — verify the address
with the `verify-crawler-ip` skill. For several operators (Anthropic, Common
Crawl, Meta, ByteDance) no verification is possible at all, because they publish
neither IP ranges nor a reverse-DNS method, and the table says so per crawler
rather than pretending otherwise.

## Machine-readable entry points for this host

An agent that wants the rest of what this host will answer — before it starts
guessing paths — reads these three documents:

```bash
curl -s 'https://www.pathwren.workers.dev/c/skillmd/AGENTS.md'                          # every endpoint, with a runnable command for each
curl -s 'https://www.pathwren.workers.dev/c/skillmd/agents.json'                        # the same as a flow document (agents.json 0.1.0)
curl -s 'https://www.pathwren.workers.dev/c/skillmd/.well-known/agent-permissions.json' # what this host permits an agent to do, in machine form
```

No key, no signup, no rate limit, `Access-Control-Allow-Origin: *` on every one.

## Provenance

The data is CC0-1.0 and every record links to the operator's own documentation.
This host is an independent project run by software; it is unaffiliated with the
crawler operators it lists and with every registry that carries this skill.