Site Crawler Stinger
You are equipping site-crawler-wasp-drone, part of the Website Auditor by Legion Code Inc.
plugin. Full scope and acceptance criteria: prd-007-site-crawler.
Purpose
Runs a platform-aware crawl of the audited site, up to 100 pages, storing raw HTML and a Markdown
extraction per page under site-data/<slug>.html / <slug>.md, plus a site-data/manifest.json
index. This Stinger's output is the shared, write-once data layer that every one of the nine
Wave-5 audit Drones (technical-seo, aeo-audit, content-semantics, internal-linking, visual-funnel,
accessibility-audit, web-security-posture, analytics-stack, performance-cwv) reads read-only, with
zero write contention between them, per PRD-007's shared workspace contract.
When to use
- Wave W4, sync, immediately after
stack-fingerprint-wasp-drone has written
_shared/target-profile.json (this Drone's stack-type dependency, per the build plan's dependency
graph and PRD-007's Depends on: prd-003)
- Any re-crawl after a mid-engagement site restructure is suspected (full re-crawl, not a patch,
see
guides/05-politeness-and-scope-limits.md)
- Never invoke this Stinger for a single-page check; that is
stack-fingerprint-stinger's scope
When not to use
- Not for classifying the site's technology stack. That is
stack-fingerprint-stinger, which runs
first and writes the target-profile.json this Stinger reads.
- Not for analyzing what was crawled (SEO, AEO, accessibility, security, performance, content,
linking, funnel, or analytics findings). This Stinger only fetches and stores; every Wave-5 Drone
owns its own analysis of
site-data/.
- Not for crawling authenticated or gated areas, submitting forms, or exceeding 100 pages without
an explicit user-approved re-run. See PRD-007's Non-Goals.
Procedure
- Confirm
_shared/target-profile.json exists with non-null platform, rendering, and
confidence. If missing, stop and hand back to the orchestrating agent.
- Read the platform value and select the seed-path strategy for it
(
guides/02-platform-traversal-strategies.md).
- Run
shared/scripts/crawl-extract.py (or its algorithm) to crawl up to 100 same-domain pages,
breadth-first, respecting robots.txt and rate limits
(guides/01-crawl-procedure.md, guides/05-politeness-and-scope-limits.md).
- Store each page as
site-data/<slug>.html + site-data/<slug>.md using the deterministic
slugify algorithm, and write site-data/manifest.json as the single index
(guides/04-storage-and-manifest-convention.md).
- Record every unreachable URL in the manifest's
unreachable[] array with a reason; never retry
into a block (guides/03-fetching-and-rendering.md).
- Write a short run summary into this engagement's
_shared/run-ledger.json entry for this Drone
(pages fetched, pages unreachable, platform strategy used). Do not write into this plugin
repository's own library/.
References map
references/templates/manifest-schema.md, load when building or consuming manifest.json and
the exact field contract is needed.
references/templates/site-data-manifest.example.json, load as a worked example of the manifest
shape before writing or parsing one.
references/research/distilled-site-crawler.md, load when a fetching/rendering/platform-
detection claim needs verification or a dispute needs settling.
- Source research archive (not shipped with the plugin): the shipped distillation summarizes it; do not require it.
references/scripts/README.md and shared/scripts/crawl-extract.py, load/run to actually
execute the crawl.
guides/01-crawl-procedure.md through guides/05-politeness-and-scope-limits.md, load per-verb
as described in each guide's own heading.
Related drones and stingers
- stack-fingerprint-stinger - runs before this Stinger in wave
W1a, writes
_shared/target-profile.json, this Stinger's binding platform-classification input.
- technical-seo-stinger - one of nine Wave-5 Drones reading
site-data/
read-only; consult when a manifest or storage-convention question is actually a technical-SEO
scope question instead.
- aeo-audit-stinger - another Wave-5 reader of
site-data/.
- content-semantics-stinger - another Wave-5 reader of
site-data/, the primary consumer of the .md extraction quality caveats in
guides/03-fetching-and-rendering.md.
- internal-linking-stinger - another Wave-5 reader of
site-data/,
and the most likely consumer of manifest.json's unreachable[] array as evidence.
- visual-funnel-stinger, accessibility-audit-stinger, web-security-posture-stinger, analytics-stack-stinger, performance-cwv-stinger - the remaining four of the nine Wave-5 Drones reading
site-data/ read-only.
- keyword-intelligence-stinger - runs one wave earlier (W3, not
W4) and does NOT read
site-data/, since it has not been written yet at that point in the
sequence; see that Stinger's guides/03-tier-3-ai-inference.md for how it independently sources
content instead.
Critical Directive
- You must read all files and context contained within your skill.
- In the event your core knowledge does not provide sufficient guidance you must make every attempt to search the internet, related knowledge base documentation files, and other available resources to supplement your knowledge prior to proceeding with your task.
- Additional related skills can be found here:
Ship Gate
Ship Gate not applicable to this Stinger's own runtime procedure. This Stinger's output
(site-data/) is written into the external customer's audit workspace
(www.<domain>-audit/site-data/), not into this plugin repository's own tracked source, so a
site-crawler run produces no committable code inside this repo for security-stinger,
quality-stinger, or github-repo-health-stinger to gate. The Ship Gate does apply, unmodified, to
any change made to this plugin repository's own files, including this SKILL.md, its guides, its
references, and shared/scripts/crawl-extract.py, before that change is committed and pushed.
1---2name: site-crawler-stinger-23description: "Platform-aware crawl to depth 100, raw HTML+MD storage in site-data/, which nine Wave-5 Drones then read read-only with zero write contention."4license: AGPL-3.0-or-later5compatibility: "Claude Code, Cursor, ChatGPT Codex, Claude Cowork."6metadata:7 source: https://skillmd.com/skills/legioncodeinc/site-crawler-stinger-28 hive-tier: stinger9 hive-drone: site-crawler-wasp-drone10 research-window: "2026-08-18 (single sweep)"11 primary-surface: external-website-audit12---1314# Site Crawler Stinger1516You are equipping **site-crawler-wasp-drone**, part of the Website Auditor by Legion Code Inc.17plugin. Full scope and acceptance criteria: [prd-007-site-crawler](../../library/requirements/backlog/prd-007-site-crawler/prd-007-site-crawler-index.md).1819## Purpose2021Runs a platform-aware crawl of the audited site, up to 100 pages, storing raw HTML and a Markdown22extraction per page under `site-data/<slug>.html` / `<slug>.md`, plus a `site-data/manifest.json`23index. This Stinger's output is the shared, write-once data layer that every one of the nine24Wave-5 audit Drones (technical-seo, aeo-audit, content-semantics, internal-linking, visual-funnel,25accessibility-audit, web-security-posture, analytics-stack, performance-cwv) reads read-only, with26zero write contention between them, per PRD-007's shared workspace contract.2728## When to use2930- Wave W4, sync, immediately after `stack-fingerprint-wasp-drone` has written31 `_shared/target-profile.json` (this Drone's stack-type dependency, per the build plan's dependency32 graph and PRD-007's `Depends on: prd-003`)33- Any re-crawl after a mid-engagement site restructure is suspected (full re-crawl, not a patch,34 see `guides/05-politeness-and-scope-limits.md`)35- Never invoke this Stinger for a single-page check; that is `stack-fingerprint-stinger`'s scope3637## When not to use3839- Not for classifying the site's technology stack. That is `stack-fingerprint-stinger`, which runs40 first and writes the `target-profile.json` this Stinger reads.41- Not for analyzing what was crawled (SEO, AEO, accessibility, security, performance, content,42 linking, funnel, or analytics findings). This Stinger only fetches and stores; every Wave-5 Drone43 owns its own analysis of `site-data/`.44- Not for crawling authenticated or gated areas, submitting forms, or exceeding 100 pages without45 an explicit user-approved re-run. See PRD-007's Non-Goals.4647## Procedure48491. Confirm `_shared/target-profile.json` exists with non-null `platform`, `rendering`, and50 `confidence`. If missing, stop and hand back to the orchestrating agent.512. Read the platform value and select the seed-path strategy for it52 (`guides/02-platform-traversal-strategies.md`).533. Run `shared/scripts/crawl-extract.py` (or its algorithm) to crawl up to 100 same-domain pages,54 breadth-first, respecting `robots.txt` and rate limits55 (`guides/01-crawl-procedure.md`, `guides/05-politeness-and-scope-limits.md`).564. Store each page as `site-data/<slug>.html` + `site-data/<slug>.md` using the deterministic57 slugify algorithm, and write `site-data/manifest.json` as the single index58 (`guides/04-storage-and-manifest-convention.md`).595. Record every unreachable URL in the manifest's `unreachable[]` array with a reason; never retry60 into a block (`guides/03-fetching-and-rendering.md`).616. Write a short run summary into this engagement's `_shared/run-ledger.json` entry for this Drone62 (pages fetched, pages unreachable, platform strategy used). Do not write into this plugin63 repository's own `library/`.6465## References map6667- `references/templates/manifest-schema.md`, load when building or consuming `manifest.json` and68 the exact field contract is needed.69- `references/templates/site-data-manifest.example.json`, load as a worked example of the manifest70 shape before writing or parsing one.71- `references/research/distilled-site-crawler.md`, load when a fetching/rendering/platform-72 detection claim needs verification or a dispute needs settling.73- Source research archive (not shipped with the plugin): the shipped distillation summarizes it; do not require it.74- `references/scripts/README.md` and `shared/scripts/crawl-extract.py`, load/run to actually75 execute the crawl.76- `guides/01-crawl-procedure.md` through `guides/05-politeness-and-scope-limits.md`, load per-verb77 as described in each guide's own heading.7879## Related drones and stingers8081- [stack-fingerprint-stinger](../stack-fingerprint-stinger) - runs before this Stinger in wave82 W1a, writes `_shared/target-profile.json`, this Stinger's binding platform-classification input.83- [technical-seo-stinger](../technical-seo-stinger) - one of nine Wave-5 Drones reading `site-data/`84 read-only; consult when a manifest or storage-convention question is actually a technical-SEO85 scope question instead.86- [aeo-audit-stinger](../aeo-audit-stinger) - another Wave-5 reader of `site-data/`.87- [content-semantics-stinger](../content-semantics-stinger) - another Wave-5 reader of88 `site-data/`, the primary consumer of the `.md` extraction quality caveats in89 `guides/03-fetching-and-rendering.md`.90- [internal-linking-stinger](../internal-linking-stinger) - another Wave-5 reader of `site-data/`,91 and the most likely consumer of `manifest.json`'s `unreachable[]` array as evidence.92- [visual-funnel-stinger](../visual-funnel-stinger), [accessibility-audit-stinger](../accessibility-audit-stinger), [web-security-posture-stinger](../web-security-posture-stinger), [analytics-stack-stinger](../analytics-stack-stinger), [performance-cwv-stinger](../performance-cwv-stinger) - the remaining four of the nine Wave-5 Drones reading `site-data/` read-only.93- [keyword-intelligence-stinger](../keyword-intelligence-stinger) - runs one wave earlier (W3, not94 W4) and does NOT read `site-data/`, since it has not been written yet at that point in the95 sequence; see that Stinger's `guides/03-tier-3-ai-inference.md` for how it independently sources96 content instead.9798## Critical Directive99100- You must read all files and context contained within your skill.101- In the event your core knowledge does not provide sufficient guidance you must make every attempt to search the internet, related knowledge base documentation files, and other available resources to supplement your knowledge prior to proceeding with your task.102- Additional related skills can be found here:103 - [site-crawler-wasp-drone](../../agents/site-crawler-wasp-drone.md) - this Stinger's paired Drone.104 - [stack-fingerprint-stinger](../stack-fingerprint-stinger) - upstream dependency, writes `target-profile.json`.105106## Ship Gate107108Ship Gate not applicable to this Stinger's own runtime procedure. This Stinger's output109(`site-data/`) is written into the external customer's audit workspace110(`www.<domain>-audit/site-data/`), not into this plugin repository's own tracked source, so a111site-crawler run produces no committable code inside this repo for security-stinger,112quality-stinger, or github-repo-health-stinger to gate. The Ship Gate does apply, unmodified, to113any change made to this plugin repository's own files, including this SKILL.md, its guides, its114references, and `shared/scripts/crawl-extract.py`, before that change is committed and pushed.