How do I install the Verify Crawler Ip skill?

Run npx skillmds add pathwren/verify-crawler-ip in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Verify Crawler Ip safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: executes scripts, makes network calls. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Verify Crawler Ip?

Any agent that reads SKILL.md files can use this skill, including Claude Code, Claude.ai, Cursor, Codex, Windsurf, and 60+ others. Install it into the agent's skills directory and it loads on demand.

Is Verify Crawler Ip free to use?

Yes. Installing skills from SkillMD is free. This skill is licensed under CC0-1.

Who published Verify Crawler Ip?

Pathwren (@pathwren) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Verify Crawler Ip by pathwren — Agent Skill

Name: Verify Crawler Ip
Author: Pathwren

Verify Crawler Ip

Decide whether a request that claims to be GPTBot, ClaudeBot, Googlebot or another named crawler really came from that operator, by checking the client address against the operator's own published IP ranges - and find out which operators publish no ranges at all, where a miss proves nothing. Use before blocking, allowlisting or rate-limiting traffic on the strength of a user-agent header.

Pathwren 3.7 KB Updated Sep 3, 2026

File contents

verify-crawler-ip

When to use this

Something claims to be a named crawler and the decision you are about to make — allowlist it, block it, bill it, exclude it from analytics — should not rest on a header anyone can spoof. This checks the address against the ranges the operator publishes, mirrored here from 15 operator endpoints and refreshed every six hours.

Check one address

curl -s 'https://www.pathwren.workers.dev/c/skillmd/tools/verify-crawler?ip=66.249.66.1&ua=Googlebot'

answer.results[].verdict is in-a-published-range or not, matches[] names the prefix, the source and the operator_source_url the prefix came from, so the finding can be checked against the operator rather than against us. Several addresses at once: repeat &ip=.

Find out whether verification is even possible

curl -s 'https://www.pathwren.workers.dev/c/skillmd/tools/verification-methods?crawler=claudebot'

verification_method is published-ranges, reverse-dns or none, with how spelling out the procedure. Ask this FIRST for any crawler you are about to treat a miss as evidence about.

The raw ranges

curl -s 'https://www.pathwren.workers.dev/c/skillmd/ip-ranges/all.txt'                  # every CIDR, one per line
curl -s 'https://www.pathwren.workers.dev/c/skillmd/ip-ranges/all.json'                 # the same, with sources
curl -s 'https://www.pathwren.workers.dev/c/skillmd/ip-ranges/google-googlebot.json'    # one operator source
curl -s 'https://www.pathwren.workers.dev/c/skillmd/data/ip-sources.json'               # the endpoints, and when each was last read
curl -s 'https://www.pathwren.workers.dev/c/skillmd/status.json'                        # were those endpoints up on the last refresh?

The limit of the answer — read this before you act on a miss

A published prefix is the operator's network, not a signature. A match means the address is inside a range the operator publishes; it does not authenticate the request, so check the user-agent too.

Absence proves nothing. Most crawlers in the index publish no ranges at all, and several document reverse DNS instead. Treating "not in a published range" as "forged" will block real crawlers — call verification-methods before concluding anything from a miss, and check status.json: a range file that failed to refresh is a stale answer, and it says so.

Machine-readable entry points for this host

An agent that wants the rest of what this host will answer — before it starts guessing paths — reads these three documents:

curl -s 'https://www.pathwren.workers.dev/c/skillmd/AGENTS.md'                          # every endpoint, with a runnable command for each
curl -s 'https://www.pathwren.workers.dev/c/skillmd/agents.json'                        # the same as a flow document (agents.json 0.1.0)
curl -s 'https://www.pathwren.workers.dev/c/skillmd/.well-known/agent-permissions.json' # what this host permits an agent to do, in machine form

No key, no signup, no rate limit, Access-Control-Allow-Origin: * on every one.

Provenance

The data is CC0-1.0 and every record links to the operator's own documentation. This host is an independent project run by software; it is unaffiliated with the crawler operators it lists and with every registry that carries this skill.

---name: verify-crawler-ipdescription: Decide whether a request that claims to be GPTBot, ClaudeBot, Googlebot or another named crawler really came from that operator, by checking the client address against the operator's own published IP ranges - and find out which operators publish no ranges at all, where a miss proves nothing. Use before blocking, allowlisting or rate-limiting traffic on the strength of a user-agent header.license: CC0-1.0---# verify-crawler-ip## When to use thisSomething claims to be a named crawler and the decision you are about to make —allowlist it, block it, bill it, exclude it from analytics — should not rest on aheader anyone can spoof. This checks the address against the ranges the operatorpublishes, mirrored here from 15 operator endpoints and refreshed everysix hours.## Check one address```bashcurl -s 'https://www.pathwren.workers.dev/c/skillmd/tools/verify-crawler?ip=66.249.66.1&ua=Googlebot'````answer.results[].verdict` is `in-a-published-range` or not, `matches[]` namesthe `prefix`, the `source` and the `operator_source_url` the prefix came from,so the finding can be checked against the operator rather than against us.Several addresses at once: repeat `&ip=`.## Find out whether verification is even possible```bashcurl -s 'https://www.pathwren.workers.dev/c/skillmd/tools/verification-methods?crawler=claudebot'````verification_method` is `published-ranges`, `reverse-dns` or `none`, with `how`spelling out the procedure. Ask this FIRST for any crawler you are about totreat a miss as evidence about.## The raw ranges```bashcurl -s 'https://www.pathwren.workers.dev/c/skillmd/ip-ranges/all.txt' # every CIDR, one per linecurl -s 'https://www.pathwren.workers.dev/c/skillmd/ip-ranges/all.json' # the same, with sourcescurl -s 'https://www.pathwren.workers.dev/c/skillmd/ip-ranges/google-googlebot.json' # one operator sourcecurl -s 'https://www.pathwren.workers.dev/c/skillmd/data/ip-sources.json' # the endpoints, and when each was last readcurl -s 'https://www.pathwren.workers.dev/c/skillmd/status.json' # were those endpoints up on the last refresh?```## The limit of the answer — read this before you act on a miss**A published prefix is the operator's network, not a signature.** A match meansthe address is inside a range the operator publishes; it does not authenticatethe request, so check the user-agent too.**Absence proves nothing.** Most crawlers in the index publish no ranges at all,and several document reverse DNS instead. Treating "not in a published range" as"forged" will block real crawlers — call `verification-methods` before concludinganything from a miss, and check `status.json`: a range file that failed torefresh is a stale answer, and it says so.## Machine-readable entry points for this hostAn agent that wants the rest of what this host will answer — before it startsguessing paths — reads these three documents:```bashcurl -s 'https://www.pathwren.workers.dev/c/skillmd/AGENTS.md' # every endpoint, with a runnable command for eachcurl -s 'https://www.pathwren.workers.dev/c/skillmd/agents.json' # the same as a flow document (agents.json 0.1.0)curl -s 'https://www.pathwren.workers.dev/c/skillmd/.well-known/agent-permissions.json' # what this host permits an agent to do, in machine form```No key, no signup, no rate limit, `Access-Control-Allow-Origin: *` on every one.## ProvenanceThe data is CC0-1.0 and every record links to the operator's own documentation.This host is an independent project run by software; it is unaffiliated with thecrawler operators it lists and with every registry that carries this skill.

Frequently asked questions