Site Crawler Stinger

Site Crawler Stinger

Forge status: stages 1-6 complete (Topic, Research, Distillation, References, Guides, final Skill/Bee authorship). Stage 7 (Register: beekeeper-suit roster entry, deploy, cross-repo reference sync) has not run yet. Every factual claim below traces to references/research/raw/ via references/research/distilled-site-crawler.md, or is flagged explicitly as a judgment call where that archive is thin. See the honesty notes throughout, especially in guides/02-platform-traversal-strategies.md and guides/04-storage-and-manifest-convention.md, both of which name specific claims this archive does not ground.

You are equipping site-crawler-worker-bee, part of the Website Auditor by Legion Code Inc. plugin. Full scope and acceptance criteria: prd-007-site-crawler.

Purpose

Runs a platform-aware crawl of the audited site, up to 100 pages, storing raw HTML and a Markdown extraction per page under site-data/<slug>.html / <slug>.md, plus a site-data/manifest.json index. This Stinger's output is the shared, write-once data layer that every one of the nine Wave-5 audit Bees (technical-seo, aeo-audit, content-semantics, internal-linking, visual-funnel, accessibility-audit, web-security-posture, analytics-stack, performance-cwv) reads read-only, with zero write contention between them, per PRD-007's shared workspace contract.

When to use

• Wave W4, sync, immediately after stack-fingerprint-worker-bee has written _shared/target-profile.json (this Bee's stack-type dependency, per the build plan's dependency graph and PRD-007's Depends on: prd-003)
• Any re-crawl after a mid-engagement site restructure is suspected (full re-crawl, not a patch, see guides/05-politeness-and-scope-limits.md)
• Never invoke this Stinger for a single-page check; that is stack-fingerprint-stinger's scope

When not to use

• Not for classifying the site's technology stack. That is stack-fingerprint-stinger, which runs first and writes the target-profile.json this Stinger reads.
• Not for analyzing what was crawled (SEO, AEO, accessibility, security, performance, content, linking, funnel, or analytics findings). This Stinger only fetches and stores; every Wave-5 Bee owns its own analysis of site-data/.
• Not for crawling authenticated or gated areas, submitting forms, or exceeding 100 pages without an explicit user-approved re-run. See PRD-007's Non-Goals.

Procedure

1. Confirm _shared/target-profile.json exists with non-null platform, rendering, and confidence. If missing, stop and hand back to the orchestrating agent.
2. Read the platform value and select the seed-path strategy for it (guides/02-platform-traversal-strategies.md).
3. Run shared/scripts/crawl-extract.py (or its algorithm) to crawl up to 100 same-domain pages, breadth-first, respecting robots.txt and rate limits (guides/01-crawl-procedure.md, guides/05-politeness-and-scope-limits.md).
4. Store each page as site-data/<slug>.html + site-data/<slug>.md using the deterministic slugify algorithm, and write site-data/manifest.json as the single index (guides/04-storage-and-manifest-convention.md).
5. Record every unreachable URL in the manifest's unreachable[] array with a reason; never retry into a block (guides/03-fetching-and-rendering.md).
6. Write a short run summary into this engagement's _shared/run-ledger.json entry for this Bee (pages fetched, pages unreachable, platform strategy used). Do not write into this plugin repository's own library/.

References map

• references/templates/manifest-schema.md, load when building or consuming manifest.json and the exact field contract is needed.
• references/templates/site-data-manifest.example.json, load as a worked example of the manifest shape before writing or parsing one.
• references/research/distilled-site-crawler.md, load when a fetching/rendering/platform- detection claim needs verification or a dispute needs settling.
• references/research/raw/, load when tracing a distilled claim back to its primary source.
• references/scripts/README.md and shared/scripts/crawl-extract.py, load/run to actually execute the crawl.
• guides/01-crawl-procedure.md through guides/05-politeness-and-scope-limits.md, load per-verb as described in each guide's own heading.

Related bees and stingers

• stack-fingerprint-stinger - runs before this Stinger in wave W1a, writes _shared/target-profile.json, this Stinger's binding platform-classification input.
• technical-seo-stinger - one of nine Wave-5 Bees reading site-data/ read-only; consult when a manifest or storage-convention question is actually a technical-SEO scope question instead.
• aeo-audit-stinger - another Wave-5 reader of site-data/.
• content-semantics-stinger - another Wave-5 reader of site-data/, the primary consumer of the .md extraction quality caveats in guides/03-fetching-and-rendering.md.
• internal-linking-stinger - another Wave-5 reader of site-data/, and the most likely consumer of manifest.json's unreachable[] array as evidence.
• visual-funnel-stinger, accessibility-audit-stinger, web-security-posture-stinger, analytics-stack-stinger, performance-cwv-stinger - the remaining four of the nine Wave-5 Bees reading site-data/ read-only.
• keyword-intelligence-stinger - runs one wave earlier (W3, not W4) and does NOT read site-data/, since it has not been written yet at that point in the sequence; see that Stinger's guides/03-tier-3-ai-inference.md for how it independently sources content instead.

Critical Directive

• You must read all files and context contained within your skill.
• In the event your core knowledge does not provide sufficient guidance you must make every attempt to search the internet, related knowledge base documentation files, and other available resources to supplement your knowledge prior to proceeding with your task.
• Additional related skills can be found here:
  • site-crawler-worker-bee - this Stinger's paired Bee.
  • stack-fingerprint-stinger - upstream dependency, writes target-profile.json.

Ship Gate

Ship Gate not applicable to this Stinger's own runtime procedure. This Stinger's output (site-data/) is written into the external customer's audit workspace (www.<domain>-audit/site-data/), not into this plugin repository's own tracked source, so a site-crawler run produces no committable code inside this repo for security-stinger, quality-stinger, or github-repo-health-stinger to gate. The Ship Gate does apply, unmodified, to any change made to this plugin repository's own files, including this SKILL.md, its guides, its references, and shared/scripts/crawl-extract.py, before that change is committed and pushed.