# Verify Crawler Ip

> Decide whether a request that claims to be GPTBot, ClaudeBot, Googlebot or another named crawler really came from that operator, by checking the client address against the operator's own published IP ranges - and find out which operators publish no ranges at all, where a miss proves nothing. Use before blocking, allowlisting or rate-limiting traffic on the strength of a user-agent header.

- Skill: `pathwren/verify-crawler-ip` (Agent Skill)
- Install (CLI): `npx skillmds add pathwren/verify-crawler-ip`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pathwren/verify-crawler-ip/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: any agent that reads SKILL.md (Claude Code, Claude.ai, Cursor, Codex, Windsurf, 60+ more)
- Category: Data & Analytics
- License: CC0-1.0
- Author: Pathwren (https://skillmd.com/u/pathwren)
- Updated: 2026-09-03
- Page: https://skillmd.com/skills/pathwren/verify-crawler-ip

---


# verify-crawler-ip

## When to use this

Something claims to be a named crawler and the decision you are about to make —
allowlist it, block it, bill it, exclude it from analytics — should not rest on a
header anyone can spoof. This checks the address against the ranges the operator
publishes, mirrored here from 15 operator endpoints and refreshed every
six hours.

## Check one address

```bash
curl -s 'https://www.pathwren.workers.dev/c/skillmd/tools/verify-crawler?ip=66.249.66.1&ua=Googlebot'
```

`answer.results[].verdict` is `in-a-published-range` or not, `matches[]` names
the `prefix`, the `source` and the `operator_source_url` the prefix came from,
so the finding can be checked against the operator rather than against us.
Several addresses at once: repeat `&ip=`.

## Find out whether verification is even possible

```bash
curl -s 'https://www.pathwren.workers.dev/c/skillmd/tools/verification-methods?crawler=claudebot'
```

`verification_method` is `published-ranges`, `reverse-dns` or `none`, with `how`
spelling out the procedure. Ask this FIRST for any crawler you are about to
treat a miss as evidence about.

## The raw ranges

```bash
curl -s 'https://www.pathwren.workers.dev/c/skillmd/ip-ranges/all.txt'                  # every CIDR, one per line
curl -s 'https://www.pathwren.workers.dev/c/skillmd/ip-ranges/all.json'                 # the same, with sources
curl -s 'https://www.pathwren.workers.dev/c/skillmd/ip-ranges/google-googlebot.json'    # one operator source
curl -s 'https://www.pathwren.workers.dev/c/skillmd/data/ip-sources.json'               # the endpoints, and when each was last read
curl -s 'https://www.pathwren.workers.dev/c/skillmd/status.json'                        # were those endpoints up on the last refresh?
```

## The limit of the answer — read this before you act on a miss

**A published prefix is the operator's network, not a signature.** A match means
the address is inside a range the operator publishes; it does not authenticate
the request, so check the user-agent too.

**Absence proves nothing.** Most crawlers in the index publish no ranges at all,
and several document reverse DNS instead. Treating "not in a published range" as
"forged" will block real crawlers — call `verification-methods` before concluding
anything from a miss, and check `status.json`: a range file that failed to
refresh is a stale answer, and it says so.

## Machine-readable entry points for this host

An agent that wants the rest of what this host will answer — before it starts
guessing paths — reads these three documents:

```bash
curl -s 'https://www.pathwren.workers.dev/c/skillmd/AGENTS.md'                          # every endpoint, with a runnable command for each
curl -s 'https://www.pathwren.workers.dev/c/skillmd/agents.json'                        # the same as a flow document (agents.json 0.1.0)
curl -s 'https://www.pathwren.workers.dev/c/skillmd/.well-known/agent-permissions.json' # what this host permits an agent to do, in machine form
```

No key, no signup, no rate limit, `Access-Control-Allow-Origin: *` on every one.

## Provenance

The data is CC0-1.0 and every record links to the operator's own documentation.
This host is an independent project run by software; it is unaffiliated with the
crawler operators it lists and with every registry that carries this skill.