# Geo Crawl

> Thin pointer to the ReadableByAI GEO crawl probe — whether AI crawlers can reach, fetch, and read a site before any citation work. Does not reimplement the probe.

- Skill: `abouchard11/geo-crawl` (Agent Skill)
- Install (CLI): `npx skillmds@latest add abouchard11/geo-crawl`
- Raw SKILL.md: https://api.skillmd.com/api/skills/abouchard11/geo-crawl/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: abouchard11 (https://skillmd.com/u/abouchard11)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/abouchard11/geo-crawl

---


# GEO Crawl — input gate

This skill does **not** own the probe. The engine, bot registry, log parser, and flag playbook live in [geo-crawl-audit](https://github.com/abouchard11/geo-crawl-audit) (ReadableByAI, [readablebyai.com](https://readablebyai.com)). Prefer engine SHA `de557923` or later (PR #3, 2026-08-31): RFC 9309 Allow/Disallow tie-break, failed-baseline scoring, CSR_SHELL scoped to the baseline body. The tie-break was a real test failure. It flipped no live `/geo-crawl` STOP in practice.

Do not copy `geo_probe.py`, `drain_parser.py`, or `bots.json` into this repo. Do not invent a second robots.txt checklist. Run that skill, then translate the flags into a STOP / CONTINUE verdict for the rest of this suite.

**First:** Read `~/.claude/skills/seo-references/core.md`.

**Canonical skill:** if `geo-crawl-audit/skill/SKILL.md` is on disk (sibling clone or `~/.claude/skills/geo-crawl-audit`), follow that file for Mode A and for the origin-log parser. This file is only the mount point and the suite handshake. Prefer the **public** `geo-crawl-audit` skill over any private clone. Private clones can lag on honesty-clause wording.

Public scan if the operator does not have the repo: https://readablebyai.com — treat it as Mode A lite.

## Planes (do not collapse)

Two log instruments. Do not feed one into the other.

| Plane | Repo | What this skill may do |
|---|---|---|
| Engine | public `geo-crawl-audit` | Run `geo_probe.py` + `drain_parser.py`. Cite flag codes from `references/interpreting.md`. |
| Mode B parser | public `drain_parser.py` | Origin-class logs only: nginx/apache/LiteSpeed combined access logs, or a raw Vercel NDJSON export that still has `proxy.userAgent` and `proxy.clientIp`. |
| Drain product | public ReadableByAI | A **different** instrument. Hosted: [readablebyai.com/logs/hosted](https://readablebyai.com/logs/hosted) → `/api/drain/<key>`. Owner receiver: `https://drain.readablebyai.com/api/drain` (GET `ok`). Events: `rba_crawler_hit` / minimized `rba_benchmark_crawler_hit`. The receiver verifies IP in memory and **discards** UA, IP, and exact path. Never pipe those events into `drain_parser.py`. Never reconstruct a UA. Do **not** POST to `https://readablebyai.com/api/drain` (410 tombstone). Never copy drain keys or secrets. |
| Drain receiver source | private `geo-bot-drain` | Contained receiver behind `drain.readablebyai.com`. Pointer only. Never copy the receiver, secrets, or `ALLOWED_DOMAINS`. |
| Evidence store | private `readablebyai-evidence` | Pointer only. Never copy `index-1/`, outreach lists, notice archives, or raw probe JSON into this repo or into Graphiti. |
| Operator artifacts | private `hq` (`outputs/geo-*`) | May cite that a dated portfolio run exists. Do not paste the run. |
| Marketplace | private `alex-private-marketplace` | Not a GEO source. Do not install or duplicate desks from it. |

Private evidence is never public. A visibility flip on the site repo must not be able to publish Index mappings or outreach kits — that is why the evidence store is split.

## Route, do not duplicate

- Citation / mention matrix → `/geo` (only after this gate)
- Passages / snippets → `/aeo`
- Preferred Sources badge → `/preferred-source`
- GSC indexing → `/indexer`
- Entity name / sameAs → `/entity`

## Step 1: Parse Domain

Extract the domain. Strip protocol, www, trailing slash, lowercase. Resolve against the MCP Routing Map. Portfolio runs iterate the roster.

If the domain is not on the routing map, Mode A may still run. Log confirmation may run only on logs the operator owns or is authorized to retain. Do not ingest third-party contributor drains.

## Step 2: Run the probe (Mode A)

Locate `scripts/geo_probe.py` from a local `geo-crawl-audit` checkout at PR #3 or later. Prefer, in order:

1. `../geo-crawl-audit/scripts/geo_probe.py` relative to this suite
2. A path the operator already has on disk
3. Clone `https://github.com/abouchard11/geo-crawl-audit` if missing, then run

```bash
python3 scripts/geo_probe.py example.com --out ./audit-results
```

Read `geo_audit_report.md` and `geo_audit.json`. Apply the honesty clause from that repo: a simulated-UA differential means a bot-sensitive layer exists, not that the real crawler is blocked.

If the probe cannot run:
- Print `[SEO] geo-crawl-audit probe unavailable — clone https://github.com/abouchard11/geo-crawl-audit or scan https://readablebyai.com.`
- Do not fake flags.
- `/geo` may continue only if the operator explicitly accepts an unprobed run. Record that in the Graphiti summary.

### Mode A identity (does not exist)

Mode A never verifies that a request is a real vendor crawler. `geo_probe.py` has no IP-range code. It sends the registry UA string from the probing machine. The operator already knows that IP is not OpenAI, Amazon, or anyone else.

| Mode A may claim | Mode A may not claim |
|---|---|
| The origin treated this UA string differently than the baseline (status, body, TTFB). | A real GPTBot / OAI-SearchBot / Amazonbot / Claude-User was blocked or allowed. |
| A bot-sensitive layer exists. | The request came from a vendor network. |

Identity verification is a Mode B property (`drain_parser.py --verify` against published ranges or pinned CIDRs) or a ReadableByAI drain property (range check in memory, then UA and IP discarded). Do not write "identity UNVERIFIABLE" on a Mode A scorecard as if it distinguished Amazonbot. It is the Mode A condition of every token.

`OAI-AdsBot` (OpenAI docs, `OAI-AdsBot/1.0`) is advertising verification for pages submitted as ChatGPT ads. Class: advertising. Registry: `category: advertising`, `probe: false`. It is not retrieval and not user_fetch. Blocking it does not STOP `/geo`.

Key `BOT_DIFFERENTIAL` on **citation surface**, not on whether the vendor publishes IPs. `/geo` measures OpenAI, Anthropic, Perplexity, Google, and Microsoft. Amazonbot grounds Alexa / Rufus, not those engines. Meta, ByteDance, and Apple tokens are the same class of off-surface retrieval. A Mode A differential on those UAs is still real (the origin treated that string differently). It does not STOP citation work.

## Step 3: Confirm when required — pick the instrument

Required when Mode A raises `BOT_DIFFERENTIAL`, `SLOW_TTFB`, or `TTFB_VARIANCE` on retrieval or user_fetch bots. Do not skip because a public scan looked clean. Name which instrument ran.

### Instrument 1 — Mode B (`drain_parser.py`)

Needs an **origin-logging** host. Combined access logs (LiteSpeed / nginx / apache) are the default class. A raw Vercel NDJSON export is origin-class only if UA and client IP are still on the row.

```bash
python3 scripts/drain_parser.py access.log --format combined --verify --out ./audit-results
python3 scripts/drain_parser.py logs/*.ndjson --out ./audit-results --verify
```

`--verify` is Mode B only. It is a no-op for bots with no published range. Do not treat `verification=unknown` as confirmed identity, and do not import that property back into a Mode A handshake.

Vercel runtime logs are not Mode B. Do not treat `vercel___get_runtime_logs` as origin logs.

LiteSpeed hosts (example: `htxpermitfix.com`) **are** the right class for this instrument. Minimized ReadableByAI drain events are the wrong class.

### Instrument 2 — ReadableByAI drain (not the parser)

Use when the host is already on the owner-portfolio drain (`drain.readablebyai.com`) or a customer-owned hosted drain. Query owned `rba_crawler_hit` or minimized `rba_benchmark_crawler_hit` events. Read `bot`, `verification`, `status_class` (and `verification_method` / `collection_basis` when present). Do not call `drain_parser.py` on those rows. Do not invent UA strings so the parser will accept them.

First owned result on this instrument (readablebyai.com, 2026-08-31, operator PostHog): CLEAR for retrieval. Verified 2xx from vendor ranges on googlebot, oai-searchbot, gptbot, chatgpt-user, bingbot, perplexitybot. A Googlebot-shaped population hitting api routes at 100% 4xx is a scanner, not an incident — do not escalate it. Own `geo_probe` runs can land in the same dataset as impostors. Neither flips CLEAR.

### Owned-log rule

1. Origin combined / raw NDJSON the operator exported — Instrument 1.
2. ReadableByAI drain events the operator already owns — Instrument 2.
3. Never raw rows from `readablebyai-evidence`.
4. Never a drain configured as **Projects: All**. Receiver excludes itself.
5. Never the tombstone at `readablebyai.com/api/drain` (HTTP 410).

No origin log and no owned RBA events: print **UNCONFIRMED**. STOP `/geo` only when the flag is CRITICAL on a retrieval or user_fetch bot whose surface `/geo` measures (OpenAI, Anthropic, Perplexity, Google, Microsoft). Amazon / Meta / ByteDance / Apple differentials stay CONTINUE WITH FIXES on citation-surface grounds.

## Step 4: Suite handshake

Issue one verdict. Use flag codes from `geo-crawl-audit/references/interpreting.md` — do not redefine them. Say which log instrument ran, or that neither did.

| Verdict | When | What the rest of the suite may do |
|---|---|---|
| **STOP** | `CSR_SHELL`; `BOT_DIFFERENTIAL` on retrieval or user_fetch for bots whose surface `/geo` measures (OpenAI, Anthropic, Perplexity, Google, Microsoft), even UNCONFIRMED; `ROBOTS_BLOCKS` on those same tokens from engine SHA `de557923`+; baseline not a normal 200 (inconclusive) | No prompt matrix. No passage rewrite sold as GEO. Fix the gate. Re-probe. |
| **CONTINUE WITH FIXES** | `SLOW_TTFB` / `TTFB_VARIANCE`; `THIN_HTML`; `NO_SITEMAP` / `NO_ROBOTS`; training-only `ROBOTS_BLOCKS`; `BOT_DIFFERENTIAL` on Amazon / Meta / ByteDance / Apple tokens | `/geo` and `/aeo` may run. First Recommended Action is the probe fix or a Mode B origin-log pull, not a new blog. |
| **CLEAR** | Score 85–100, no CRITICAL flags on the `/geo` citation surface — or Instrument 2 shows verified 2xx on those tokens | `/geo` proceeds normally. |

Training-bot blocks (`GPTBot`, `ClaudeBot`, `Google-Extended`, …) are a rights choice. They do not by themselves STOP citation work. Say so.
Advertising-class tokens (`OAI-AdsBot`) do not STOP citation work.

`NO_LLMS_TXT` is a footnote. Never lead with it.

Use engine SHA `de557923`+ for new `ROBOTS_BLOCKS` reads. The pre-PR #3 tie-break flipped no live STOP in the 2026-08-31 field pair.

## Step 5: Output

Follow the Output Protocol from core.md:
1. Print the scorecard worst-first, then the STOP / CONTINUE / CLEAR verdict
2. Extract structured summary (include flag codes, which log instrument ran, owned vs missing)
3. Draft Graphiti save with name `GEO Crawl — [domain]`, `group_id` from the MCP Routing Map. Suite methodology facts use `group_id=midnight-seo-skills`. Live `add_memory` is human/policy-closed. Episodes already in the write-ahead spool must not be re-drafted.

Do not paste a prompt-probe matrix here. That is `/geo` after CLEAR or CONTINUE WITH FIXES.
Do not paste Index probe bodies, notice lists, or drain secrets into the summary.