Geo Crawl

GEO Crawl — input gate

This skill does not own the probe. The engine, bot registry, log parser, and flag playbook live in geo-crawl-audit (ReadableByAI, readablebyai.com). Prefer engine SHA de557923 or later (PR #3, 2026-08-31): RFC 9309 Allow/Disallow tie-break, failed-baseline scoring, CSR_SHELL scoped to the baseline body. The tie-break was a real test failure. It flipped no live /geo-crawl STOP in practice.

Do not copy geo_probe.py, drain_parser.py, or bots.json into this repo. Do not invent a second robots.txt checklist. Run that skill, then translate the flags into a STOP / CONTINUE verdict for the rest of this suite.

First: Read ~/.claude/skills/seo-references/core.md.

Canonical skill: if geo-crawl-audit/skill/SKILL.md is on disk (sibling clone or ~/.claude/skills/geo-crawl-audit), follow that file for Mode A and for the origin-log parser. This file is only the mount point and the suite handshake. Prefer the public geo-crawl-audit skill over any private clone. Private clones can lag on honesty-clause wording.

Public scan if the operator does not have the repo: https://readablebyai.com — treat it as Mode A lite.

Planes (do not collapse)

Two log instruments. Do not feed one into the other.

Private evidence is never public. A visibility flip on the site repo must not be able to publish Index mappings or outreach kits — that is why the evidence store is split.

Route, do not duplicate

• Citation / mention matrix → /geo (only after this gate)
• Passages / snippets → /aeo
• Preferred Sources badge → /preferred-source
• GSC indexing → /indexer
• Entity name / sameAs → /entity

Step 1: Parse Domain

Extract the domain. Strip protocol, www, trailing slash, lowercase. Resolve against the MCP Routing Map. Portfolio runs iterate the roster.

If the domain is not on the routing map, Mode A may still run. Log confirmation may run only on logs the operator owns or is authorized to retain. Do not ingest third-party contributor drains.

Step 2: Run the probe (Mode A)

Locate scripts/geo_probe.py from a local geo-crawl-audit checkout at PR #3 or later. Prefer, in order:

1. ../geo-crawl-audit/scripts/geo_probe.py relative to this suite
2. A path the operator already has on disk
3. Clone https://github.com/abouchard11/geo-crawl-audit if missing, then run

python3 scripts/geo_probe.py example.com --out ./audit-results

Read geo_audit_report.md and geo_audit.json. Apply the honesty clause from that repo: a simulated-UA differential means a bot-sensitive layer exists, not that the real crawler is blocked.

If the probe cannot run:

• Print [SEO] geo-crawl-audit probe unavailable — clone https://github.com/abouchard11/geo-crawl-audit or scan https://readablebyai.com.
• Do not fake flags.
• /geo may continue only if the operator explicitly accepts an unprobed run. Record that in the Graphiti summary.

Mode A identity (does not exist)

Mode A never verifies that a request is a real vendor crawler. geo_probe.py has no IP-range code. It sends the registry UA string from the probing machine. The operator already knows that IP is not OpenAI, Amazon, or anyone else.

Identity verification is a Mode B property (drain_parser.py --verify against published ranges or pinned CIDRs) or a ReadableByAI drain property (range check in memory, then UA and IP discarded). Do not write "identity UNVERIFIABLE" on a Mode A scorecard as if it distinguished Amazonbot. It is the Mode A condition of every token.

OAI-AdsBot (OpenAI docs, OAI-AdsBot/1.0) is advertising verification for pages submitted as ChatGPT ads. Class: advertising. Registry: category: advertising, probe: false. It is not retrieval and not user_fetch. Blocking it does not STOP /geo.

Key BOT_DIFFERENTIAL on citation surface, not on whether the vendor publishes IPs. /geo measures OpenAI, Anthropic, Perplexity, Google, and Microsoft. Amazonbot grounds Alexa / Rufus, not those engines. Meta, ByteDance, and Apple tokens are the same class of off-surface retrieval. A Mode A differential on those UAs is still real (the origin treated that string differently). It does not STOP citation work.

Step 3: Confirm when required — pick the instrument

Required when Mode A raises BOT_DIFFERENTIAL, SLOW_TTFB, or TTFB_VARIANCE on retrieval or user_fetch bots. Do not skip because a public scan looked clean. Name which instrument ran.

Instrument 1 — Mode B (drain_parser.py)

Needs an origin-logging host. Combined access logs (LiteSpeed / nginx / apache) are the default class. A raw Vercel NDJSON export is origin-class only if UA and client IP are still on the row.

python3 scripts/drain_parser.py access.log --format combined --verify --out ./audit-results
python3 scripts/drain_parser.py logs/*.ndjson --out ./audit-results --verify

--verify is Mode B only. It is a no-op for bots with no published range. Do not treat verification=unknown as confirmed identity, and do not import that property back into a Mode A handshake.

Vercel runtime logs are not Mode B. Do not treat vercel___get_runtime_logs as origin logs.

LiteSpeed hosts (example: htxpermitfix.com) are the right class for this instrument. Minimized ReadableByAI drain events are the wrong class.

Instrument 2 — ReadableByAI drain (not the parser)

Use when the host is already on the owner-portfolio drain (drain.readablebyai.com) or a customer-owned hosted drain. Query owned rba_crawler_hit or minimized rba_benchmark_crawler_hit events. Read bot, verification, status_class (and verification_method / collection_basis when present). Do not call drain_parser.py on those rows. Do not invent UA strings so the parser will accept them.

First owned result on this instrument (readablebyai.com, 2026-08-31, operator PostHog): CLEAR for retrieval. Verified 2xx from vendor ranges on googlebot, oai-searchbot, gptbot, chatgpt-user, bingbot, perplexitybot. A Googlebot-shaped population hitting api routes at 100% 4xx is a scanner, not an incident — do not escalate it. Own geo_probe runs can land in the same dataset as impostors. Neither flips CLEAR.

Owned-log rule

1. Origin combined / raw NDJSON the operator exported — Instrument 1.
2. ReadableByAI drain events the operator already owns — Instrument 2.
3. Never raw rows from readablebyai-evidence.
4. Never a drain configured as Projects: All. Receiver excludes itself.
5. Never the tombstone at readablebyai.com/api/drain (HTTP 410).

No origin log and no owned RBA events: print UNCONFIRMED. STOP /geo only when the flag is CRITICAL on a retrieval or user_fetch bot whose surface /geo measures (OpenAI, Anthropic, Perplexity, Google, Microsoft). Amazon / Meta / ByteDance / Apple differentials stay CONTINUE WITH FIXES on citation-surface grounds.

Step 4: Suite handshake

Issue one verdict. Use flag codes from geo-crawl-audit/references/interpreting.md — do not redefine them. Say which log instrument ran, or that neither did.

Training-bot blocks (GPTBot, ClaudeBot, Google-Extended, …) are a rights choice. They do not by themselves STOP citation work. Say so. Advertising-class tokens (OAI-AdsBot) do not STOP citation work.

NO_LLMS_TXT is a footnote. Never lead with it.

Use engine SHA de557923+ for new ROBOTS_BLOCKS reads. The pre-PR #3 tie-break flipped no live STOP in the 2026-08-31 field pair.

Step 5: Output

Follow the Output Protocol from core.md:

1. Print the scorecard worst-first, then the STOP / CONTINUE / CLEAR verdict
2. Extract structured summary (include flag codes, which log instrument ran, owned vs missing)
3. Draft Graphiti save with name GEO Crawl — [domain], group_id from the MCP Routing Map. Suite methodology facts use group_id=midnight-seo-skills. Live add_memory is human/policy-closed. Episodes already in the write-ahead spool must not be re-drafted.

Do not paste a prompt-probe matrix here. That is /geo after CLEAR or CONTINUE WITH FIXES. Do not paste Index probe bodies, notice lists, or drain secrets into the summary.