stardust:extract
Crawl an existing website, parse each page, extract the brand surface,
and produce a stardust-formatted snapshot of the current state under
stardust/current/. The output describes what the site is; later
sub-commands consume it to decide what it should be.
This skill is descriptive: it does not invent direction, it does not
critique, and it does not modify the live site. It writes only under
stardust/current/ and updates stardust/state.json.
Inputs
<url> — required. The origin to crawl. Examples: https://example.com,
https://example.com/shop. A path narrows the same-origin crawl to
that subtree.
--cap <N> — optional. Override the default 5-page cap. The cap
is intentionally small — a 5-page sample (home + four IA
pillars/templates) is enough for cross-page brand aggregation,
system-component detection, and the brand-review HTML; lift it
(e.g. --cap 25) when a deeper crawl is genuinely needed.
--all — optional. Lift the cap entirely; extract every
discovered page after junk filtering. Equivalent to --cap 0.
Use when the user spontaneously asks for a full crawl.
--pages <slug,slug,...> — optional. Restrict the crawl to specific
paths (slugs derived per reference/ia-extraction.md). Bypasses
the cap.
--refresh <slug> — optional. Re-extract one page that already exists
in state.json.
--single — optional. Equivalent to --cap 1. Useful for testing.
--wait <fast|medium|spec|auto> — optional. Wait strategy per page.
Default medium. See reference/playwright-recipe.md § Wait modes.
--no-junk-filter — optional. Disable the default junk-page filter
in discovery (see reference/ia-extraction.md § Filtering).
--no-consent-dismiss — optional. Skip the pre-flight consent /
cookie banner dismissal (see reference/playwright-recipe.md
§ Pre-flight: consent dismissal). Use when the redesign scope
includes the consent surface or the dismissal's side-effects
(script activation that wouldn't otherwise run) must be
avoided. Default is to dismiss, keeping screenshots, voice
aggregation, and per-section style unpolluted by the banner.
--dynamics — optional, migration-bound. Record per-page reach
signals of the dynamic surface (data endpoints, forms, modal
triggers, player ids) in each page JSON dynamic section and roll
them up in _crawl-log.json#dynamicSurface. Set by
prepare-migration, replica and migrate's safety net; never by a
bare extract, uplift or audit — dynamics is a migration concern.
Depth and classification belong to the stardust dynamics skill.
--concurrency <n> — optional. Parallel browser contexts for the
per-page capture loop. Default 4; sane range 4–8. See
§ Concurrency.
--brand-source <url> — optional, repeatable. An additional
same-brand origin whose brand surface enriches the primary
extraction (shallow capture: home + up to 2 nav-linked pages).
See § Cross-site brand sources.
--design-source <url> — optional. Design-donor origin: its
design system is captured to stardust/canon-source/ and becomes
the fixed redesign target while the primary origin supplies
content. See § Cross-site brand sources.
--prep — optional. Run in migrate-prep mode: lift the cap,
type each page, detect module candidates, capture typed content
slots, emit the prep summary. See § Prep mode below. Typically
invoked via the prepare-migration orchestrator skill rather
than directly.
Setup
Run the master skill's setup procedure first
(skills/stardust/SKILL.md § Setup): impeccable dep check, context
loader, state read.
Additional checks for this sub-command:
Playwright availability. The extraction step needs a real
browser. Detect Playwright in this order: a Playwright MCP server,
then a project-importable playwright module. The npx playwright probe is NOT sufficient — it confirms the CLI
(which resolves a global install) but the recipe and
scripts/crawl.mjs do import { chromium } from 'playwright',
and ESM module resolution does not honour a global install or
NODE_PATH — the import throws ERR_MODULE_NOT_FOUND even where
npx playwright --version succeeds. Verify the module is
import-resolvable from the project root (probe:
node -e "import('playwright').then(()=>process.exit(0))"); if it
isn't, run npm i -D playwright pixelmatch pngjs cheerio --legacy-peer-deps
(devDependencies, never --no-save — #125) or use the Playwright MCP
server, before crawling. The
--legacy-peer-deps flag is required on aem-boilerplate targets
(their pinned eslint@8 makes a plain npm i exit ERESOLVE
before playwright is even considered). Don't trust the CLI
probe alone.
--no-save installs are ephemeral. Any later real npm i (e.g. a setup step
adding a devDependency) prunes non-manifest packages, silently
removing playwright mid-pipeline. Every downstream skill that
renders (prototype, migrate, deploy, diff) must re-run the
import-resolvability probe — and re-install on failure — at the
start of its own run, not assume extract's install survived.
Script location matters. ESM resolves import 'playwright'
from the script's directory and the plugin tree ships no
node_modules: copy every script byte-identical into the project
(stardust/scripts/…) and run the copy.
Bundled crawler. skills/extract/scripts/crawl.mjs is a
runnable reference implementation of this whole sub-command —
browser config + bot-management fallback, consent dismissal, wait
- scroll, the capture list, per-page full-page screenshots
(
assets/screenshots/<slug>.png, consumed by the Phase 2.5
vision gate), response validation, and the §
Capture-hygiene hardening (visibility filter, interstitial drop,
SPA-shell flag, modal textContent capture, tracking-pixel
discounting, cross-page duplicate detection). Prefer invoking it over
hand-rolling a Playwright script per run; extend its in-page
capture() to cover any recipe field it doesn't yet emit.
Origin collision. If stardust/state.json already records
site.originUrl and the new <url> is a different origin, stop and
ask before clobbering. Stardust does not silently mix two sites in
one project.
Flow guard (migration asks only). If the ask carries migration
intent ("migrate", "to EDS", "re-platform", "1:1", "replica") and
stardust/state.json exists — or is about to be created — without
flow, hand back to the master skill § Two migration flows before
crawling: the flow is chosen and stamped there, and a keep-design
ask enters through replica (which invokes this skill with --prep
itself). A bare extract <url> for a redesign, audit or uplift is
unaffected. (Recorded: extract on a raw URL as the entry of a
same-design migration; the agent then built its own importer beside
replica.)
Browser contexts. Open a fresh BrowserContext per capture
worker (§ Concurrency; default 4). Run the consent dismissal
pre-flight per reference/playwright-recipe.md § Pre-flight:
consent dismissal unless --no-consent-dismiss is set.
Cookies persist within a context but not across contexts —
re-establish consent state per context (re-run the dismissal on
the worker's first page, or clone the probe context's
storageState). Record the resolved method in
_crawl-log.json#consent.method.
Bot-management probe. When the first navigation in the run
returns ERR_HTTP2_PROTOCOL_ERROR or ERR_QUIC_PROTOCOL_ERROR,
or hangs through the entire hard-cap on what should be a fast
origin, do not retry headless — switch to headed Chrome per
§ Failure modes → Bot-management block and record the switch in
_crawl-log.json#discovery.fetchTechnique so re-runs start in
headed mode without rediscovering the issue.
Procedure
Phase 1 — Discovery
Discover the page inventory before crawling. Procedure in
reference/ia-extraction.md. In summary:
Fetch <origin>/sitemap.xml, then <origin>/sitemap_index.xml,
then check robots.txt for Sitemap: directives.
If no sitemap is reachable, run a same-origin BFS crawl from
<url>, depth-limited to 3, link-extracting from rendered HTML.
Filter the discovered URL list: same origin only, exclude
mailto:, tel:, anchor-only links, query-only variations,
common asset paths (.css, .js, .pdf, image extensions).
De-duplicate trailing-slash variations.
Apply the junk-page filter (reference/ia-extraction.md §
Junk-page filter) unless --no-junk-filter is set. Surface the
filtered list to the user as overridable.
Apply the cap (default 5, or --cap, or --all for no cap)
and proceed silently. Print an informational summary of
what was kept and what was cut — but do not gate on user
confirmation. Users who want different scope set it
spontaneously at command time:
$stardust extract https://example.com # default 5 pages
$stardust extract https://example.com --cap 25 # bump to 25
$stardust extract https://example.com --all # lift the cap
$stardust extract https://example.com --pages home,about,pricing
$stardust extract https://example.com --single # just the entry URL
The agent reads spontaneous scope intent from the user's prompt
(e.g. "extract all pages", "look at just the home and pricing",
"do a full crawl") and applies the equivalent flag. No
re-confirmation needed once intent is clear.
Informational output (not a prompt — proceed immediately):
Discovered 38 pages on https://example.com (sitemap.xml).
Filtered as likely junk (5): /test/, /sample-page/, /holiday1/, ...
Selecting 5 highest-priority pages:
- / (home)
- /about
- /pricing
- /products
- /contact
Cut (28 pages, --all to lift): /blog/post-1, /blog/post-2, ...
Extracting...
Selection heuristic: page-type checklist first, then score-based
ranking (home + IA-pillar keywords + sitemap priority − archive /
version markers). See reference/ia-extraction.md § Page
selection and § Priority for the cap. The English-only keyword
list is a known limitation for localized sites.
Write the discovered list to stardust/current/_crawl-log.json
(created if absent) with _provenance and the full discovery
reasoning, including filteredAsJunk[] and userChoice. This is
an audit trail, not a state file.
Phase 2 — Per-page extraction
For each page in the cap-respecting list, render with Playwright
following reference/playwright-recipe.md. Captures run
concurrently per § Concurrency. The recipe is
mandatory per page — in particular, do not skip the wait, scroll, or
capture-list steps:
- Viewport 1440 × 900 @ 2× DPR
- Wait per the configured wait mode (default
medium; see § Wait
modes in reference/playwright-recipe.md)
- Disable animations via
prefers-reduced-motion: reduce
- After the wait resolves, scroll to bottom in 4 viewport-height
steps with 300 ms pauses, then return to top — this is required
to trigger lazy-load and IntersectionObserver-driven content
- Record
waitMs and waitMode in the per-page _provenance
Capture per page (full schema in reference/current-state-schema.md):
Page metadata (title, meta description, OG tags, theme-color)
Semantic structure: heading outline, landmark roles, sections
Hero headline + lede (resolved) — heroHeadline / heroLede
picked by font-size × hero-region with a junk/hidden-state filter and
a clean meta-description fallback (per reference/playwright-recipe.md
§ Capture list 5-bis). Required for JS-rendered sites whose
document-order headings surface modal / promo / count junk
instead of the real tagline.
Content: visible text per section (full innerText, no
truncation per reference/playwright-recipe.md § Capture
list 7), structured paragraphs (body[]), lists, FAQ Q/A
pairs, and review/testimonial quotes per
§ Capture list 7-bis. Without these structured fields,
every body region under a heading falls back to placeholder
signature at migrate time.
CTA labels and href targets, link inventory (internal vs external)
Per-section computed style summary: dominant colors, font families
in use, spacing rhythm, border-radius, shadows
Media inventory: img with currentSrc/srcset captured with
query strings intact plus a resolves flag (HEAD/GET with browser
UA + Referer), intrinsic dimensions, inline SVG count, video/iframe
presence, cssBackgrounds[] (including pseudo-element ::before/
::after walks per § Capture list 11) so background-image
heroes and motifs do not silently disappear and 404ing CDN
images are flagged before migrate ships about:error.
Font files captured via network-intercept (per § Capture list
16): every woff2/woff/ttf/otf response saved under
assets/fonts/ and recorded in _brand-extraction.json#type.files[]
with licensing flag.
Icon-font detection (per § Capture list 17): when the page
uses [class^="icon-"] with non-default ::before
font-family + codepoint, capture the family, save the file,
and record the iconClass → codepoint table in
_brand-extraction.json#iconFont.
Interactive elements: forms (with field types), buttons, modals
detected by ARIA roles
Full-page screenshot to assets/screenshots/<slug>.png —
script-captured by the bundled crawler after the wait/scroll
settle (viewport-only fallback on extremely tall pages; mode in
_signals.screenshotMode, relative path in the page JSON
screenshot field)
Dynamic surface (only with --dynamics) — per-page reach
signals: endpoints, third-party script hosts, forms, modal triggers,
player ids, hydration hints, in the page JSON dynamic section and
_crawl-log.json#dynamicSurface (schema in
reference/current-state-schema.md § Dynamic). Evidence only; the
stardust dynamics sub-skill probes archetypes in depth and decides.
Save to stardust/current/pages/<slug>.json with _provenance as the
first key. The bundled crawler also saves the settled rendered DOM
verbatim as stardust/current/pages/<slug>.html (page.content()
after the wait/scroll settle; path in the record's renderedHtml
field). Capture once, parse offline: every downstream importer or
sibling generator iterates its extraction against this artifact —
free, reproducible, and provenance — instead of re-running live
probes per selector guess (recorded: 4+ live round-trips per page
family before the switch). Live probes stay for what the static DOM
cannot answer: geometry and computed styles. Save referenced media to stardust/current/assets/media/
preserving basename plus a short content hash.
Live-render evidence (synthesis is forbidden). Refuse to mark
a page extracted in state.json unless its _provenance
contains renderedBy: "playwright", an ISO-8601 fetchedAt, a
positive integer waitMs, a waitMode from the recipe, and a
final httpStatus in the 2xx/3xx range. These five fields are
the contract enforced by reference/current-state-schema.md
§ Live-render evidence and read back by every downstream phase
via validateProvenance() per
skills/stardust/reference/state-machine.md § Provenance
validation. Synthesizing a page record from
_brand-extraction.json plus URL patterns plus captured photos
— the 2026-04-30 e-commerce shortcut — is the failure mode this
guard exists to prevent. When the agent (or a delegated sub-
agent) cannot satisfy the contract for a page, treat the page
as a Phase 2 failure: record under _crawl-log.json#crawl.failures[]
with errorClass: "ProvenanceMissing" and continue.
Mark the page extracted in state.json immediately after each
successful page write. If a page fails, record the error in
_crawl-log.json and continue — extraction is best-effort per page.
Phase 2.5 — Vision verification
Before anything downstream is authored, look at each captured
page's screenshot (assets/screenshots/<slug>.png) — the multimodal
model reads the image — and verify it against the extracted record:
- Does the recorded hero (headline + asset) match what the pixels show?
- Is the extracted palette plausible against the pixels?
- Does a
cssBackgrounds: [] record look believable, or is imagery
visibly present — a silent capture failure?
- Is the logo captured?
- Is the page actually rendered — not a consent wall, bot-block
page, or blank SPA shell?
Read the whole page through its thumbnail —
node stardust/scripts/thumb.mjs stardust/current/assets/screenshots/<slug>.png --width 480
(skills/extract/scripts/thumb.mjs, copied beside crawl.mjs; a
box-filter downscale, so 1-px rules and hairline borders survive; its
--max-bytes cap, 150 KB by default, re-encodes a heavy page narrower
first and crops only as a last resort, never below 60 % of the page) —
never the full-resolution screenshot, and read any detail as a crop of
the full capture. The check is of the WHOLE page: read the script's
stdout line before the image, and when it prints a share below 100
(cropped at <n>px of <H> = <share>%) run the same command with
--offset <n> and read <slug>-thumb-<n>.png too, repeating while its
rows <n>-<end>px line ends short of <H>. A recorded hands-off run's
thumbnails kept a median 28 % of each page and the check read them as
the page. A thumbnail enters the context only after thumb.mjs has
written it (≤ 150 KB, or the floor thumbnail it named on stderr, exit 1)
— or the vision check runs in a subagent that reads the thumbnails and
returns one line per page
(<slug>: ok | recaptured | suspect — <note>); a recorded hands-off run
read eight *-thumb.png files of 268–608 KB each into one context —
3.2 MB of pixels for eight verdicts.
On mismatch, re-run that page's capture with the escalation ladder
before proceeding: bump the wait mode one step
(reference/playwright-recipe.md § Wait modes), then headed Chrome
(§ Bot-management fallback), then a fresh browser context. Record
the outcome per page in _crawl-log.json#visionCheck[]:
{ "slug": "pricing", "verdict": "recaptured", "notes": "record said zero CSS backgrounds; screenshot shows a full-bleed photo hero" }
verdict is "ok" | "recaptured" | "suspect" — suspect means the
mismatch survived the ladder; downstream phases treat that record as
unreliable. Vision is the authoritative capture check; the
heuristic defenses (low-media flag, spaShellSuspect, duplicate
hash) remain as cheap early signals but no longer gate alone.
Phase 3 — Brand-surface extraction
Run after the capture phases (2–2.5). Aggregation may proceed
incrementally as concurrent page captures complete (§ Concurrency),
but the written file must reflect every extracted page — including
brand-source pages per § Cross-site brand sources.
Produces stardust/current/_brand-extraction.json
per reference/brand-surface.md. Some fields are home-only (logo,
voice samples, register heuristic); the visual tokens that drive
DESIGN.md (palette, radius, shadow, type) are aggregated across all
extracted pages to avoid the home-page bias documented in
brand-surface.md § Aggregation scope.
Computed-style census — run the shipped instrument; author no
probe. node <plugin>/skills/extract/scripts/style-census.mjs
(copied into the project like crawl.mjs, § Setup) measures every
captured page at 1440 and writes
stardust/current/_computed-styles.json; add --width 360 when the
breakpoints include it. One more live pass over every page — run it
ONCE, after the crawl, in the background; it opens the live side
through live-session.mjs like every other live instrument. Read the
palette, type, motif and hover values from the file's aggregate and
cite each from its sources[] (url, selector, prop); the
per-page detail sits under pages[url][width].
Content-cap model — run the shipped probe; author no lift (#124).
node stardust/scripts/replica/cap-probe.mjs <archetype-url>… --write-design stardust/current/DESIGN.json (replica's script; copy it with live-session.mjs)
renders each archetype at 1440 and at a DERIVED wide width —
max(2560, largest cap × 1.25); a probe AT a cap's width cannot see it — and
records every box that stops growing (kind shell / content / module,
px, authored via, tier) into extensions.breakpoints: containerMaxWidth
(the --max-width source; null when fluid), probeWidth, caps[],
modules[]. One navigation per archetype after the census (never beside
it), once DESIGN.json is authored (it merges). A cap on the shell or
on every archetype is design intent whatever its value; capRegister lists
single-module or archetype-divergent caps for the inconsistency register.
The cap model never records layout breakpoints: a redesign target keeps its own two;
replica keeps the source's, or maps them to --target-breakpoints.
Captures:
- Logo by the v1 priority chain: inline SVG →
<img> with
logo-ish class/id → apple-touch-icon → og:image → favicon →
synthesized placeholder. Save to stardust/current/assets/logo.<ext>.
- Favicon — ALWAYS captured as its own asset (independent of the
logo chain) to
stardust/current/assets/favicon.<ext>, per
reference/playwright-recipe.md § Favicon capture. Downstream,
prototype embeds it in the proposed page head and deploy ships
it to the Edge Delivery site.
- Palette — aggregate computed colors across all extracted
pages (background, text, accents, borders, hovers). Frequency-sort,
cluster near-duplicates, emit a role-named list (background, surface,
text, primary, secondary, accent).
- Type — font families in use with their weights, sizes, and
computed line-heights. Identify the heading family vs body family.
Run the modular-scale audit (
brand-surface.md § Modular-scale
audit) and emit scaleAudit.kind = "modular" | "ad-hoc".
- Motifs — signature border-radius (cross-page mode of non-zero
values, weighted by element count), shadow stack (top 3 distinct,
cross-page), gradient inventory, common patterns (chip, badge,
card, hero-with-image). When the home-only mode disagrees with the
cross-page mode, surface the divergence in
_provenance.notes.
- Voice samples — first paragraph of body copy, the hero headline,
3 representative CTA labels, a representative link list. Used by
direct later but extracted now so the network round-trip is over.
- Hero image — elevate the home page's primary visual
asset to
voice.heroImage (per reference/brand-surface.md
§ heroImage resolution), so downstream prototype picks the
live hero rather than the og:image from the raw media list.
- Hero medium (signature) — when the hero/first viewport carries a
moving asset (background
<video> / HLS / canvas / WebGL /
Lottie / animated SVG / scroll-driven motion), elevate it to
voice.heroMedium (per reference/brand-surface.md § heroMedium
resolution). This is the page's signature; without elevation
downstream prototype flattens it to a static hero. A non-null
heroMedium triggers signature
preservation (skills/stardust/reference/intent-dimensions.md
§ 8b) at prototype time.
- Icon font — when detected per
reference/playwright-recipe.md
§ Capture list 17, populate _brand-extraction.json#iconFont
with family, file path, and the iconClass → codepoint
table so prototypes can render the brand's actual icons.
- System components — cross-page repeated DOM blocks (site
header, site footer, cross-promo strips, persistent CTAs,
breadcrumbs). Detected by heading-sequence + CTA-label fingerprint
per
reference/brand-surface.md § System components. Required —
these are usually the most load-bearing surfaces and must not
silently disappear from the redesign target.
- Origins — one entry per contributing origin in
_brand-extraction.json#origins[] per reference/brand-surface.md
§ Origins. Single-origin runs emit a one-entry array; brand-source
runs attribute widened evidence per origin.
Do not invent values. Every captured value cites a source selector or
URL in _brand-extraction.json for traceability.
Phase 4 — Seed stardust/current/PRODUCT.md and DESIGN.md
The current-state PRODUCT.md and DESIGN.md are descriptive, not
authored — there is no interview to run because the user is not
defining intent here, the agent is describing the existing site. Write
them directly using impeccable's format specs (impeccable's skill
directory is what node <plugin>/skills/stardust/scripts/impeccable-version-check.mjs --where
prints; read each spec by section, not whole):
- For PRODUCT.md, follow the section structure in impeccable's
reference/init.md § Write PRODUCT.md. File order: stardust's
<!-- stardust:provenance … --> block first (per
../stardust/reference/artifact-map.md § Provenance, as for every
file this skill writes), then # Product, then the
<!-- impeccable:product-schema 1 --> comment verbatim, then
Platform (web), Users, Product Purpose, Positioning,
Capabilities and Constraints, Brand Commitments, Evidence on Hand, Product Principles, Accessibility & Inclusion; omit a
section rather than pad it. Under Brand Commitments record the
register guess from the brand surface (sites that read as
marketing/landing → brand; tools/dashboards → product; ambiguous
→ brand with a note), the observed brand personality and the
observed anti-references. Evidence on Hand lists what was captured
under stardust/current/ with paths. Populate Users, Product Purpose, Positioning and Product Principles from the captured
copy and the brand surface. Where the agent must infer, mark the
section with _provenance: inferred and a one-line basis sentence.
- For DESIGN.md and DESIGN.json, follow the format spec in
impeccable's
reference/document.md. Populate frontmatter
(colors, typography, rounded, spacing, components) from
the captured tokens. The extensions block of DESIGN.json carries
v1's componentStyle, motifs, and voice arrays so nothing is
lost, and breakpoints — the cap-probe sizing model — verbatim.
Stardust does not invoke $impeccable init (formerly teach) or
$impeccable document for the current-state files: those commands write to project
root (the target) and run an interview. Stardust authors the
descriptive snapshot directly. The format spec from impeccable is the
contract; the runtime command is not.
The target-state PRODUCT.md and DESIGN.md at the project root are
written by $stardust direct in Phase 2 of the pipeline, not here.
Phase 5 — Render stardust/current/brand-review.html
After Phase 4 writes the descriptive PRODUCT.md and DESIGN.md, emit
the current-state brand review per
reference/brand-review-template.md.
The brand-review HTML is the first surface a human can eyeball to
verify the extraction before committing to a redesign direction —
misreads that are invisible in JSON (a wrong dominant radius, a
missing system component, a single-page palette bias) are obvious to
the eye while they are still cheap to fix (re-extract is fast;
re-direct + re-prototype is not).
The template is mandatory. In particular:
- Run the Tensions detectors listed in
reference/brand-review-template.md § Detectors. Each rule is
mechanical; emit a tension card whenever the trigger condition
matches. The review may ship with zero tensions if the data is
too thin to evaluate, but the detectors must always be run.
- Render in the brand's own captured colors and fonts, not a
stardust shell.
- Embed all CSS; do not load external JavaScript or fonts unless
the live site already does.
- Cite the source artifact for every section (e.g.
_brand-extraction.json § type under Typography).
If the data for a section is missing, omit the section — do not
fabricate placeholders. The coverage callout at the top reflects what
is missing.
Phase 6 — Update state and report
After all Phase 2-5 writes succeed:
Update stardust/state.json (schema in
skills/stardust/reference/state-machine.md):
site.originUrl, site.extractedAt, site.pageCap,
site.totalDiscovered, site.crawled
pages[] — one entry per crawled page with status: "extracted",
filled currentStatePath, empty prototypePath and migratedPath
designSource — only when --design-source was used:
{ url, capturedAt, path } per § Cross-site brand sources
Print a one-screen summary:
Extracted https://example.com (5/38 pages, sitemap.xml)
stardust/current/
PRODUCT.md, DESIGN.md, DESIGN.json, brand-review.html,
pages/ (5), assets/logo.svg, _brand-extraction.json, _crawl-log.json
Per-page evidence:
slug live waitMode waitMs status media(img/bg)
/ yes medium 2380 200 38/6
/pricing yes medium 1940 200 9/0 ⚠ low-media
/contact yes domcontentloaded(fb) 8000 200 3/0
...
Wait summary: 4 resolved at medium (avg 2.4s), 1 fallback (timed out at 8s)
→ /contact may be under-captured; consider --refresh
Media summary: 1 page flagged low-media (/pricing) — see media-coverage check
Vision check: 4 ok, 1 recaptured (/pricing) — _crawl-log.json#visionCheck
Open stardust/current/brand-review.html to verify the extraction
before running $stardust direct.
Coverage note: extracted 5 of 38 discovered pages; 5 pages
covering distinct templates is usually sufficient for the
cross-page brand surface. Widen with --cap <N> or --pages.
Next: $stardust direct (resolve a redesign direction)
The per-page evidence table is mandatory. The live column
is yes when _provenance.renderedBy === "playwright" AND
waitMs > 0, else no. A no row means the page record was
not produced by a live Playwright render — the visible column
is the defense-in-depth signal for the failure mode the
write-time guard exists to prevent (2026-04-30 e-commerce run). A
maintainer scanning the summary should see yes on every row.
Compute the wait summary by grouping each page's _provenance.waitMode
and averaging waitMs. List slugs whose waitMode ends in
(fallback) (rendered as (fb) in the table for width) as
candidates for --refresh.
The media(img/bg) column is the analogous defense-in-depth
signal for imagery. It prints <count of media.imgs> / <count of media.cssBackgrounds>. Flag a row ⚠ low-media when the page
reads as brand/marketing (register brand, or a landing/solution/
product template) yet has cssBackgrounds: [] and few large
rasters (no media.imgs entry with intrinsic width ≥ 600). That
combination is the signature of a silently-failed background /
lazy-media walk — a capture pass that specs the full background
walk (playwright-recipe.md § Capture list 11) yet silently
produces nothing still ships an image-less capture (2026-06-26
a SaaS site: cssBackgrounds: [] on every page, all product
imagery lost). A flagged row is the cue to re-run that page with
--refresh (and, if it persists, to fall back to headed Chrome per
§ Bot-management fallback). A maintainer scanning the summary should
treat a brand-register site with all-zero bg counts as suspect,
not as "this site uses no background images."
Cross-site brand sources
Two flags widen extraction beyond the primary origin — read
reference/cross-site-sources.md in full whenever either flag is
present. Both are opt-in; without them this section is inert.
Core contract (merge rules and capture shapes in the reference):
--brand-source <url> (repeatable) — a same-brand sibling
gets a shallow capture (home + ≤2 nav-linked pages, full recipe +
provenance) under stardust/current/brand-sources/<host>/.
Evidence, not inventory: never enters state.json.pages[].
Its palette/type/motif/voice evidence aggregates into
_brand-extraction.json with per-origin attribution
(origins[]); conflicts resolve toward the primary; widened
traits are attributed, never invented. Excluded from system
components, voiceTable, and cross-promo detection.
--design-source <url> — a design donor is captured to
stardust/canon-source/ (same shapes as current/, default cap),
its descriptive DESIGN.md/json derived per Phase 4 rules, and
state.json.designSource = { url, capturedAt, path: "stardust/canon-source/" } stamped. direct pins the donor
system as the target (its § Mode A); donor evidence never
aggregates into the primary _brand-extraction.json.
Sibling-site discovery
After the primary crawl (Phases 2–2.5), harvest candidate same-brand
origins from evidence already captured — no extra navigation:
- footer / nav links out to other properties
- "our brands" / "our companies" pages
hreflang alternates on other domains
- subdomain families (
shop., careers., country subdomains)
og:site_name matches across captured pages
List candidates with confidence + the evidence line in the crawl
report and in _crawl-log.json#siblingCandidates[]:
{ "origin": "https://example.co.uk", "confidence": "high", "evidence": "hreflang alternate on 4/5 pages", "decision": "included" }
- Interactive: propose — "found 3 candidate sibling properties —
include as
--brand-source?" — and proceed on the answer.
- Hands-off (
state.json.handsOff is true): auto-include up to
2 high-confidence candidates as brand-sources; record the
decision in siblingCandidates[].decision.
Discovery is capped and cheap: harvesting reads captured evidence
only, and each included sibling gets the shallow brand-source
capture (≤ 3 pages). It must never balloon the crawl.
Outputs
| Path |
Purpose |
stardust/current/PRODUCT.md |
Descriptive strategy of the existing site (impeccable format) |
stardust/current/DESIGN.md |
Descriptive visual system (Stitch format) |
stardust/current/DESIGN.json |
Sidecar with extensions for motifs, voice, components |
stardust/current/brand-review.html |
Self-contained visual review of the extraction (first eyeball-able artifact) |
stardust/current/pages/<slug>.json |
Per-page parsed structure + content |
stardust/current/pages/<slug>.html |
Settled rendered DOM (crawler sidecar; parse offline, never re-scrape) |
stardust/current/assets/logo.<ext> |
Extracted logo |
stardust/current/assets/favicon.<ext> |
Site favicon (first-class asset; prototype head + deploy consume it) |
stardust/current/assets/media/ |
Extracted media referenced by pages |
stardust/current/assets/screenshots/ |
Per-page full-page screenshots, script-captured by crawl.mjs (Phase 2.5 vision gate + brand-review) |
stardust/current/_brand-extraction.json |
Consolidated brand surface (palette, type, motifs, voice, system components) |
stardust/current/_crawl-log.json |
Discovery + crawl audit trail (incl. visionCheck[], siblingCandidates[]; dynamicSurface reach roll-up only with --dynamics) |
stardust/current/brand-sources/<host>/ |
Shallow same-brand captures (only with --brand-source) |
stardust/canon-source/ |
Design-donor capture + descriptive DESIGN.md/json (only with --design-source) |
stardust/state.json |
Updated with site + per-page status (+ designSource stamp) |
Concurrency
Page captures run concurrently: the Phase 2 queue is drained by
4–8 parallel browser contexts (--concurrency, default 4). Each
worker owns its BrowserContext; consent state is re-established per
context (Setup step 3). Media resolves / HEAD checks are batched
with Promise.all. Brand-surface aggregation (Phase 3) may proceed
incrementally as pages complete, so long as the written
_brand-extraction.json reflects every extracted page. The bundled
crawler implements the pool (crawl.mjs --concurrency <n>).
Across processes, per state-machine.md: stardust does not lock. Two
concurrent extracts on the same project are last-write-wins. Document
this in the user report; do not engineer around it.
Failure modes
- Network failure mid-crawl. Continue, record in
_crawl-log.json,
end with a partial state. State.json reflects only successfully
extracted pages. User can re-run; already-extracted pages are
skipped unless --refresh <slug>.
- HTTP 4xx/5xx, non-HTML content, soft-404s. Validated explicitly
per
reference/playwright-recipe.md § Response validation. Each
produces a distinct error class (HTTPError, ContentTypeError,
EmptyPageError) recorded in _crawl-log.json#crawl.failures[].
Failed pages do not appear in state.json as extracted —
they appear only in the failure log. Without this validation a 5xx
page silently lands as an empty success and propagates wrong data
to direct and prototype.
- Login wall. Do not attempt to authenticate. If the home page
redirects to a login screen, capture that one page, mark the rest as
unreachable, and ask the user how to proceed (provide cookies via
Playwright config, change the entry URL, or scope to public pages).
- Bot-management block (Akamai / Cloudflare / F5 / Imperva / CloudFront).
When the first navigation returns
ERR_HTTP2_PROTOCOL_ERROR,
ERR_QUIC_PROTOCOL_ERROR, or hangs through the hard-cap on a
TLS/H2 fingerprint check, the issue is JA3/H2 fingerprinting on
bundled-chromium-default headless mode — not auth, not network.
crawl.mjs runs the fallback itself, window-free: real Chrome
headless (channel: 'chrome', its own identity, no launch
flags) first, bundled Chromium only when Chrome is missing, one
real-Chrome relaunch with the challenge-solve window on an
entry-page block, and a wall — an edge 403 or a 200 verification
interstitial — is named within seconds in _crawl-log.json
(discovery.botBlock, per-page BotChallenge failures), never
captured as content (node crawl.mjs --help; a hand-rolled recipe
follows reference/playwright-recipe.md § Bot-management
fallback). Note for asset harvest: a page-level bot wall usually
does NOT gate assets — media/CSS/font URLs commonly return 200 to
a plain browser-UA curl even while every page navigation is
challenged (Cloudflare challenge, field-confirmed). Probe one
asset with curl BEFORE reaching for in-page-fetch machinery; the
in-page harvest is the fallback, not the default.
- JavaScript-only content. Playwright already handles this. If
the configured wait condition never fires within the mode's hard
cap (
reference/playwright-recipe.md § Wait modes), fall back to
domcontentloaded and capture what is rendered. Record the
fallback in the per-page _provenance.waitMode and surface in the
wait-summary line of the final report.
- Synthesis attempt (forbidden). When the agent (or a delegated
sub-agent) cannot run a real Playwright render for a page —
whether due to time pressure, token budget, or a tool/networ
…(truncated)
1---2name: extract-23description: Crawl an existing website (capped, multi-page) and seed stardust/current/ with PRODUCT.md, DESIGN.md, DESIGN.json, a per-page inventory, and the consolidated brand surface — the captured design system, palette, typography, motifs, and voice of the live site. Use when the user wants to analyze an existing site's design, extract or reverse-engineer its design system or brand, capture design tokens from a live site, import a website as the starting point for a redesign, capture the current state before a migration, or invokes `$stardust extract` (`/stardust:extract` in Claude Code). Trigger phrases include "analyze this site", "extract the design tokens", "capture the brand", "crawl the site", "reverse engineer the design". Not for scraping page data or content for its own sake (it captures design evidence, not datasets), and not for the redesign itself — extraction is descriptive; direction and prototyping happen downstream.4license: Apache-2.05compatibility: Requires Node 22+, Playwright with Chromium resolvable from the project, playwright-cli on PATH, and the impeccable skill (github.com/pbakaus/impeccable) installed alongside stardust.6metadata:7 source: https://skillmd.com/skills/adobe/extract-28---910# stardust:extract1112Crawl an existing website, parse each page, extract the brand surface,13and produce a stardust-formatted snapshot of the current state under14`stardust/current/`. The output describes what the site **is**; later15sub-commands consume it to decide what it **should be**.1617This skill is **descriptive**: it does not invent direction, it does not18critique, and it does not modify the live site. It writes only under19`stardust/current/` and updates `stardust/state.json`.2021## Inputs2223- `<url>` — required. The origin to crawl. Examples: `https://example.com`,24 `https://example.com/shop`. A path narrows the same-origin crawl to25 that subtree.26- `--cap <N>` — optional. Override the default 5-page cap. The cap27 is intentionally small — a 5-page sample (home + four IA28 pillars/templates) is enough for cross-page brand aggregation,29 system-component detection, and the brand-review HTML; lift it30 (e.g. `--cap 25`) when a deeper crawl is genuinely needed.31- `--all` — optional. Lift the cap entirely; extract every32 discovered page after junk filtering. Equivalent to `--cap 0`.33 Use when the user spontaneously asks for a full crawl.34- `--pages <slug,slug,...>` — optional. Restrict the crawl to specific35 paths (slugs derived per `reference/ia-extraction.md`). Bypasses36 the cap.37- `--refresh <slug>` — optional. Re-extract one page that already exists38 in `state.json`.39- `--single` — optional. Equivalent to `--cap 1`. Useful for testing.40- `--wait <fast|medium|spec|auto>` — optional. Wait strategy per page.41 Default `medium`. See `reference/playwright-recipe.md` § Wait modes.42- `--no-junk-filter` — optional. Disable the default junk-page filter43 in discovery (see `reference/ia-extraction.md` § Filtering).44- `--no-consent-dismiss` — optional. Skip the pre-flight consent /45 cookie banner dismissal (see `reference/playwright-recipe.md`46 § Pre-flight: consent dismissal). Use when the redesign scope47 includes the consent surface or the dismissal's side-effects48 (script activation that wouldn't otherwise run) must be49 avoided. Default is to dismiss, keeping screenshots, voice50 aggregation, and per-section style unpolluted by the banner.51- `--dynamics` — optional, **migration-bound**. Record per-page reach52 signals of the dynamic surface (data endpoints, forms, modal53 triggers, player ids) in each page JSON `dynamic` section and roll54 them up in `_crawl-log.json#dynamicSurface`. Set by55 `prepare-migration`, `replica` and `migrate`'s safety net; never by a56 bare extract, `uplift` or `audit` — dynamics is a migration concern.57 Depth and classification belong to the stardust `dynamics` skill.58- `--concurrency <n>` — optional. Parallel browser contexts for the59 per-page capture loop. Default 4; sane range 4–8. See60 § Concurrency.61- `--brand-source <url>` — optional, repeatable. An additional62 **same-brand** origin whose brand surface enriches the primary63 extraction (shallow capture: home + up to 2 nav-linked pages).64 See § Cross-site brand sources.65- `--design-source <url>` — optional. Design-donor origin: its66 design system is captured to `stardust/canon-source/` and becomes67 the fixed redesign target while the primary origin supplies68 content. See § Cross-site brand sources.69- `--prep` — optional. Run in **migrate-prep mode**: lift the cap,70 type each page, detect module candidates, capture typed content71 slots, emit the prep summary. See § Prep mode below. Typically72 invoked via the `prepare-migration` orchestrator skill rather73 than directly.7475## Setup7677Run the master skill's setup procedure first78(`skills/stardust/SKILL.md` § Setup): impeccable dep check, context79loader, state read.8081Additional checks for this sub-command:82831. **Playwright availability.** The extraction step needs a real84 browser. Detect Playwright in this order: a Playwright MCP server,85 then a project-importable `playwright` module. **The `npx86 playwright` probe is NOT sufficient** — it confirms the CLI87 (which resolves a global install) but the recipe and88 `scripts/crawl.mjs` do `import { chromium } from 'playwright'`,89 and ESM module resolution does **not** honour a global install or90 `NODE_PATH` — the import throws `ERR_MODULE_NOT_FOUND` even where91 `npx playwright --version` succeeds. Verify the module is92 import-resolvable from the project root (probe:93 `node -e "import('playwright').then(()=>process.exit(0))"`); if it94 isn't, run `npm i -D playwright pixelmatch pngjs cheerio --legacy-peer-deps`95 (devDependencies, never `--no-save` — #125) or use the Playwright MCP96 server, before crawling. The97 `--legacy-peer-deps` flag is required on `aem-boilerplate` targets98 (their pinned `eslint@8` makes a plain `npm i` exit `ERESOLVE`99 before playwright is even considered). Don't trust the CLI100 probe alone.101102 **`--no-save` installs are ephemeral.** Any later real `npm i` (e.g. a setup step103 adding a devDependency) prunes non-manifest packages, silently104 removing playwright mid-pipeline. Every downstream skill that105 renders (prototype, migrate, deploy, diff) must re-run the106 import-resolvability probe — and re-install on failure — at the107 start of its own run, not assume extract's install survived.108109 **Script location matters.** ESM resolves `import 'playwright'`110 from the *script's* directory and the plugin tree ships no111 `node_modules`: copy every script byte-identical into the project112 (`stardust/scripts/…`) and run the copy.113114 **Bundled crawler.** `skills/extract/scripts/crawl.mjs` is a115 runnable reference implementation of this whole sub-command —116 browser config + bot-management fallback, consent dismissal, wait117 + scroll, the capture list, per-page full-page screenshots118 (`assets/screenshots/<slug>.png`, consumed by the Phase 2.5119 vision gate), response validation, and the §120 Capture-hygiene hardening (visibility filter, interstitial drop,121 SPA-shell flag, modal `textContent` capture, tracking-pixel122 discounting, cross-page duplicate detection). Prefer invoking it over123 hand-rolling a Playwright script per run; extend its in-page124 `capture()` to cover any recipe field it doesn't yet emit.1252. **Origin collision.** If `stardust/state.json` already records126 `site.originUrl` and the new `<url>` is a different origin, stop and127 ask before clobbering. Stardust does not silently mix two sites in128 one project.129 **Flow guard (migration asks only).** If the ask carries migration130 intent ("migrate", "to EDS", "re-platform", "1:1", "replica") and131 `stardust/state.json` exists — or is about to be created — without132 `flow`, hand back to the master skill § Two migration flows before133 crawling: the flow is chosen and stamped there, and a keep-design134 ask enters through `replica` (which invokes this skill with `--prep`135 itself). A bare `extract <url>` for a redesign, audit or uplift is136 unaffected. (Recorded: `extract` on a raw URL as the entry of a137 same-design migration; the agent then built its own importer beside138 `replica`.)1393. **Browser contexts.** Open a fresh `BrowserContext` per capture140 worker (§ Concurrency; default 4). Run the **consent dismissal141 pre-flight** per `reference/playwright-recipe.md` § Pre-flight:142 consent dismissal *unless* `--no-consent-dismiss` is set.143 Cookies persist within a context but **not across contexts** —144 re-establish consent state per context (re-run the dismissal on145 the worker's first page, or clone the probe context's146 `storageState`). Record the resolved method in147 `_crawl-log.json#consent.method`.1484. **Bot-management probe.** When the first navigation in the run149 returns `ERR_HTTP2_PROTOCOL_ERROR` or `ERR_QUIC_PROTOCOL_ERROR`,150 or hangs through the entire hard-cap on what should be a fast151 origin, **do not retry headless** — switch to headed Chrome per152 § Failure modes → Bot-management block and record the switch in153 `_crawl-log.json#discovery.fetchTechnique` so re-runs start in154 headed mode without rediscovering the issue.155156## Procedure157158### Phase 1 — Discovery159160Discover the page inventory before crawling. Procedure in161`reference/ia-extraction.md`. In summary:1621631. Fetch `<origin>/sitemap.xml`, then `<origin>/sitemap_index.xml`,164 then check `robots.txt` for `Sitemap:` directives.1652. If no sitemap is reachable, run a same-origin BFS crawl from166 `<url>`, depth-limited to 3, link-extracting from rendered HTML.1673. Filter the discovered URL list: same origin only, exclude168 `mailto:`, `tel:`, anchor-only links, query-only variations,169 common asset paths (`.css`, `.js`, `.pdf`, image extensions).1704. De-duplicate trailing-slash variations.1715. Apply the junk-page filter (`reference/ia-extraction.md` §172 Junk-page filter) unless `--no-junk-filter` is set. Surface the173 filtered list to the user as overridable.1746. Apply the cap (default 5, or `--cap`, or `--all` for no cap)175 and **proceed silently**. Print an informational summary of176 what was kept and what was cut — but do **not** gate on user177 confirmation. Users who want different scope set it178 spontaneously at command time:179180 ```181 $stardust extract https://example.com # default 5 pages182 $stardust extract https://example.com --cap 25 # bump to 25183 $stardust extract https://example.com --all # lift the cap184 $stardust extract https://example.com --pages home,about,pricing185 $stardust extract https://example.com --single # just the entry URL186 ```187188 The agent reads spontaneous scope intent from the user's prompt189 (e.g. "extract all pages", "look at just the home and pricing",190 "do a full crawl") and applies the equivalent flag. No191 re-confirmation needed once intent is clear.192193 Informational output (not a prompt — proceed immediately):194195 ```196 Discovered 38 pages on https://example.com (sitemap.xml).197 Filtered as likely junk (5): /test/, /sample-page/, /holiday1/, ...198 Selecting 5 highest-priority pages:199 - / (home)200 - /about201 - /pricing202 - /products203 - /contact204205 Cut (28 pages, --all to lift): /blog/post-1, /blog/post-2, ...206207 Extracting...208 ```209210 Selection heuristic: page-type checklist first, then score-based211 ranking (home + IA-pillar keywords + sitemap priority − archive /212 version markers). See `reference/ia-extraction.md` § Page213 selection and § Priority for the cap. The English-only keyword214 list is a known limitation for localized sites.2152167. Write the discovered list to `stardust/current/_crawl-log.json`217 (created if absent) with `_provenance` and the full discovery218 reasoning, including `filteredAsJunk[]` and `userChoice`. This is219 an audit trail, not a state file.220221### Phase 2 — Per-page extraction222223For each page in the cap-respecting list, render with Playwright224following `reference/playwright-recipe.md`. Captures run225**concurrently** per § Concurrency. The recipe is226mandatory per page — in particular, do not skip the wait, scroll, or227capture-list steps:228229- Viewport 1440 × 900 @ 2× DPR230- Wait per the configured wait mode (default `medium`; see § Wait231 modes in `reference/playwright-recipe.md`)232- Disable animations via `prefers-reduced-motion: reduce`233- After the wait resolves, scroll to bottom in 4 viewport-height234 steps with 300 ms pauses, then return to top — this is required235 to trigger lazy-load and IntersectionObserver-driven content236- Record `waitMs` and `waitMode` in the per-page `_provenance`237238Capture per page (full schema in `reference/current-state-schema.md`):239240- Page metadata (title, meta description, OG tags, theme-color)241- Semantic structure: heading outline, landmark roles, sections242- **Hero headline + lede (resolved)** — `heroHeadline` / `heroLede`243 picked by font-size × hero-region with a junk/hidden-state filter and244 a clean meta-description fallback (per `reference/playwright-recipe.md`245 § Capture list 5-bis). Required for JS-rendered sites whose246 document-order headings surface modal / promo / count junk247 instead of the real tagline.248- Content: visible text per section (full innerText, **no249 truncation** per `reference/playwright-recipe.md` § Capture250 list 7), structured paragraphs (`body[]`), lists, FAQ Q/A251 pairs, and review/testimonial quotes per252 § Capture list 7-bis. Without these structured fields,253 every body region under a heading falls back to placeholder254 signature at migrate time.255- CTA labels and href targets, link inventory (internal vs external)256- Per-section computed style summary: dominant colors, font families257 in use, spacing rhythm, border-radius, shadows258- Media inventory: img with `currentSrc`/`srcset` captured **with259 query strings intact** plus a `resolves` flag (HEAD/GET with browser260 UA + Referer), intrinsic dimensions, inline SVG count, video/iframe261 presence, `cssBackgrounds[]` (including pseudo-element `::before`/262 `::after` walks per § Capture list 11) so `background-image`263 heroes and motifs do not silently disappear and 404ing CDN264 images are flagged before migrate ships `about:error`.265- Font files captured via network-intercept (per § Capture list266 16): every `woff2`/`woff`/`ttf`/`otf` response saved under267 `assets/fonts/` and recorded in `_brand-extraction.json#type.files[]`268 with licensing flag.269- Icon-font detection (per § Capture list 17): when the page270 uses `[class^="icon-"]` with non-default `::before`271 font-family + codepoint, capture the family, save the file,272 and record the `iconClass → codepoint` table in273 `_brand-extraction.json#iconFont`.274- Interactive elements: forms (with field types), buttons, modals275 detected by ARIA roles276- Full-page screenshot to `assets/screenshots/<slug>.png` —277 script-captured by the bundled crawler after the wait/scroll278 settle (viewport-only fallback on extremely tall pages; mode in279 `_signals.screenshotMode`, relative path in the page JSON280 `screenshot` field)281282- **Dynamic surface (only with `--dynamics`)** — per-page reach283 signals: endpoints, third-party script hosts, forms, modal triggers,284 player ids, hydration hints, in the page JSON `dynamic` section and285 `_crawl-log.json#dynamicSurface` (schema in286 `reference/current-state-schema.md § Dynamic`). Evidence only; the287 stardust `dynamics` sub-skill probes archetypes in depth and decides.288289Save to `stardust/current/pages/<slug>.json` with `_provenance` as the290first key. **The bundled crawler also saves the settled rendered DOM291verbatim as `stardust/current/pages/<slug>.html`** (`page.content()`292after the wait/scroll settle; path in the record's `renderedHtml`293field). Capture once, parse offline: every downstream importer or294sibling generator iterates its extraction against this artifact —295free, reproducible, and provenance — instead of re-running live296probes per selector guess (recorded: 4+ live round-trips per page297family before the switch). Live probes stay for what the static DOM298cannot answer: geometry and computed styles. Save referenced media to `stardust/current/assets/media/`299preserving basename plus a short content hash.300301**Live-render evidence (synthesis is forbidden).** Refuse to mark302a page `extracted` in `state.json` unless its `_provenance`303contains `renderedBy: "playwright"`, an ISO-8601 `fetchedAt`, a304positive integer `waitMs`, a `waitMode` from the recipe, and a305final `httpStatus` in the 2xx/3xx range. These five fields are306the contract enforced by `reference/current-state-schema.md`307§ Live-render evidence and read back by every downstream phase308via `validateProvenance()` per309`skills/stardust/reference/state-machine.md` § Provenance310validation. Synthesizing a page record from311`_brand-extraction.json` plus URL patterns plus captured photos312— the 2026-04-30 e-commerce shortcut — is the failure mode this313guard exists to prevent. When the agent (or a delegated sub-314agent) cannot satisfy the contract for a page, treat the page315as a Phase 2 failure: record under `_crawl-log.json#crawl.failures[]`316with `errorClass: "ProvenanceMissing"` and continue.317318Mark the page `extracted` in `state.json` immediately after each319successful page write. If a page fails, record the error in320`_crawl-log.json` and continue — extraction is best-effort per page.321322### Phase 2.5 — Vision verification323324Before anything downstream is authored, **look** at each captured325page's screenshot (`assets/screenshots/<slug>.png`) — the multimodal326model reads the image — and verify it against the extracted record:327328- Does the recorded hero (headline + asset) match what the pixels show?329- Is the extracted palette plausible against the pixels?330- Does a `cssBackgrounds: []` record look believable, or is imagery331 visibly present — a silent capture failure?332- Is the logo captured?333- Is the page actually rendered — not a consent wall, bot-block334 page, or blank SPA shell?335336Read the whole page through its thumbnail —337`node stardust/scripts/thumb.mjs stardust/current/assets/screenshots/<slug>.png --width 480`338(`skills/extract/scripts/thumb.mjs`, copied beside `crawl.mjs`; a339box-filter downscale, so 1-px rules and hairline borders survive; its340`--max-bytes` cap, 150 KB by default, re-encodes a heavy page narrower341first and crops only as a last resort, never below 60 % of the page) —342never the full-resolution screenshot, and read any detail as a crop of343the full capture. The check is of the WHOLE page: read the script's344stdout line before the image, and when it prints a share below 100345(`cropped at <n>px of <H> = <share>%`) run the same command with346`--offset <n>` and read `<slug>-thumb-<n>.png` too, repeating while its347`rows <n>-<end>px` line ends short of `<H>`. A recorded hands-off run's348thumbnails kept a median 28 % of each page and the check read them as349the page. **A thumbnail enters the context only after `thumb.mjs` has350written it (≤ 150 KB, or the floor thumbnail it named on stderr, exit 1)351— or the vision check runs in a subagent that reads the thumbnails and352returns one line per page**353(`<slug>: ok | recaptured | suspect — <note>`); a recorded hands-off run354read eight `*-thumb.png` files of 268–608 KB each into one context —3553.2 MB of pixels for eight verdicts.356357On mismatch, re-run that page's capture with the escalation ladder358before proceeding: bump the wait mode one step359(`reference/playwright-recipe.md` § Wait modes), then headed Chrome360(§ Bot-management fallback), then a fresh browser context. Record361the outcome per page in `_crawl-log.json#visionCheck[]`:362363```json364{ "slug": "pricing", "verdict": "recaptured", "notes": "record said zero CSS backgrounds; screenshot shows a full-bleed photo hero" }365```366367`verdict` is `"ok" | "recaptured" | "suspect"` — `suspect` means the368mismatch survived the ladder; downstream phases treat that record as369unreliable. Vision is the **authoritative** capture check; the370heuristic defenses (low-media flag, `spaShellSuspect`, duplicate371hash) remain as cheap early signals but no longer gate alone.372373### Phase 3 — Brand-surface extraction374375Run after the capture phases (2–2.5). Aggregation **may proceed376incrementally** as concurrent page captures complete (§ Concurrency),377but the written file must reflect every extracted page — including378brand-source pages per § Cross-site brand sources.379Produces `stardust/current/_brand-extraction.json`380per `reference/brand-surface.md`. Some fields are home-only (logo,381voice samples, register heuristic); the visual tokens that drive382DESIGN.md (palette, radius, shadow, type) are aggregated across **all383extracted pages** to avoid the home-page bias documented in384`brand-surface.md` § Aggregation scope.385386**Computed-style census — run the shipped instrument; author no387probe.** `node <plugin>/skills/extract/scripts/style-census.mjs`388(copied into the project like `crawl.mjs`, § Setup) measures every389captured page at 1440 and writes390`stardust/current/_computed-styles.json`; add `--width 360` when the391breakpoints include it. One more live pass over every page — run it392ONCE, after the crawl, in the background; it opens the live side393through `live-session.mjs` like every other live instrument. Read the394palette, type, motif and hover values from the file's `aggregate` and395cite each from its `sources[]` (`url`, `selector`, `prop`); the396per-page detail sits under `pages[url][width]`.397398**Content-cap model — run the shipped probe; author no lift (#124).**399`node stardust/scripts/replica/cap-probe.mjs <archetype-url>… --write-design400stardust/current/DESIGN.json` (replica's script; copy it with `live-session.mjs`)401renders each archetype at 1440 and at a DERIVED wide width —402max(2560, largest cap × 1.25); a probe AT a cap's width cannot see it — and403records every box that stops growing (kind `shell` / `content` / `module`,404px, authored via, tier) into `extensions.breakpoints`: `containerMaxWidth`405(the `--max-width` source; null when fluid), `probeWidth`, `caps[]`,406`modules[]`. One navigation per archetype after the census (never beside407it), once DESIGN.json is authored (it merges). A cap on the shell or408on every archetype is design intent whatever its value; `capRegister` lists409single-module or archetype-divergent caps for the inconsistency register.410The cap model never records layout breakpoints: a redesign target keeps its own two;411replica keeps the source's, or maps them to `--target-breakpoints`.412Captures:413414- **Logo** by the v1 priority chain: inline SVG → `<img>` with415 logo-ish class/id → `apple-touch-icon` → `og:image` → favicon →416 synthesized placeholder. Save to `stardust/current/assets/logo.<ext>`.417- **Favicon** — ALWAYS captured as its own asset (independent of the418 logo chain) to `stardust/current/assets/favicon.<ext>`, per419 `reference/playwright-recipe.md` § Favicon capture. Downstream,420 `prototype` embeds it in the proposed page head and `deploy` ships421 it to the Edge Delivery site.422- **Palette** — aggregate computed colors across **all extracted423 pages** (background, text, accents, borders, hovers). Frequency-sort,424 cluster near-duplicates, emit a role-named list (background, surface,425 text, primary, secondary, accent).426- **Type** — font families in use with their weights, sizes, and427 computed line-heights. Identify the heading family vs body family.428 Run the modular-scale audit (`brand-surface.md` § Modular-scale429 audit) and emit `scaleAudit.kind = "modular" | "ad-hoc"`.430- **Motifs** — signature border-radius (cross-page mode of non-zero431 values, weighted by element count), shadow stack (top 3 distinct,432 cross-page), gradient inventory, common patterns (chip, badge,433 card, hero-with-image). When the home-only mode disagrees with the434 cross-page mode, surface the divergence in `_provenance.notes`.435- **Voice samples** — first paragraph of body copy, the hero headline,436 3 representative CTA labels, a representative link list. Used by437 `direct` later but extracted now so the network round-trip is over.438- **Hero image** — elevate the home page's primary visual439 asset to `voice.heroImage` (per `reference/brand-surface.md`440 § heroImage resolution), so downstream prototype picks the441 live hero rather than the `og:image` from the raw media list.442- **Hero medium (signature)** — when the hero/first viewport carries a443 *moving* asset (background `<video>` / HLS / canvas / WebGL /444 Lottie / animated SVG / scroll-driven motion), elevate it to445 `voice.heroMedium` (per `reference/brand-surface.md` § heroMedium446 resolution). This is the page's **signature**; without elevation447 downstream prototype flattens it to a static hero. A non-null448 `heroMedium` triggers signature449 preservation (`skills/stardust/reference/intent-dimensions.md`450 § 8b) at prototype time.451- **Icon font** — when detected per `reference/playwright-recipe.md`452 § Capture list 17, populate `_brand-extraction.json#iconFont`453 with family, file path, and the `iconClass → codepoint`454 table so prototypes can render the brand's actual icons.455- **System components** — cross-page repeated DOM blocks (site456 header, site footer, cross-promo strips, persistent CTAs,457 breadcrumbs). Detected by heading-sequence + CTA-label fingerprint458 per `reference/brand-surface.md` § System components. Required —459 these are usually the most load-bearing surfaces and must not460 silently disappear from the redesign target.461- **Origins** — one entry per contributing origin in462 `_brand-extraction.json#origins[]` per `reference/brand-surface.md`463 § Origins. Single-origin runs emit a one-entry array; brand-source464 runs attribute widened evidence per origin.465466Do not invent values. Every captured value cites a source selector or467URL in `_brand-extraction.json` for traceability.468469### Phase 4 — Seed `stardust/current/PRODUCT.md` and `DESIGN.md`470471The current-state PRODUCT.md and DESIGN.md are **descriptive, not472authored** — there is no interview to run because the user is not473defining intent here, the agent is describing the existing site. Write474them directly using impeccable's format specs (impeccable's skill475directory is what `node <plugin>/skills/stardust/scripts/impeccable-version-check.mjs --where`476prints; read each spec by section, not whole):477478- For PRODUCT.md, follow the section structure in impeccable's479 `reference/init.md` § Write PRODUCT.md. File order: stardust's480 `<!-- stardust:provenance … -->` block first (per481 `../stardust/reference/artifact-map.md` § Provenance, as for every482 file this skill writes), then `# Product`, then the483 `<!-- impeccable:product-schema 1 -->` comment verbatim, then484 `Platform` (`web`), `Users`, `Product Purpose`, `Positioning`,485 `Capabilities and Constraints`, `Brand Commitments`, `Evidence on486 Hand`, `Product Principles`, `Accessibility & Inclusion`; omit a487 section rather than pad it. Under `Brand Commitments` record the488 register guess from the brand surface (sites that read as489 marketing/landing → `brand`; tools/dashboards → `product`; ambiguous490 → `brand` with a note), the observed brand personality and the491 observed anti-references. `Evidence on Hand` lists what was captured492 under `stardust/current/` with paths. Populate `Users`, `Product493 Purpose`, `Positioning` and `Product Principles` from the captured494 copy and the brand surface. Where the agent must infer, mark the495 section with `_provenance: inferred` and a one-line basis sentence.496- For DESIGN.md and DESIGN.json, follow the format spec in497 impeccable's `reference/document.md`. Populate frontmatter498 (`colors`, `typography`, `rounded`, `spacing`, `components`) from499 the captured tokens. The `extensions` block of DESIGN.json carries500 v1's `componentStyle`, `motifs`, and `voice` arrays so nothing is501 lost, and `breakpoints` — the cap-probe sizing model — verbatim.502503Stardust does **not** invoke `$impeccable init` (formerly `teach`) or504`$impeccable document` for the current-state files: those commands write to project505root (the *target*) and run an interview. Stardust authors the506descriptive snapshot directly. The format spec from impeccable is the507contract; the runtime command is not.508509The target-state PRODUCT.md and DESIGN.md at the project root are510written by `$stardust direct` in Phase 2 of the pipeline, not here.511512### Phase 5 — Render `stardust/current/brand-review.html`513514After Phase 4 writes the descriptive PRODUCT.md and DESIGN.md, emit515the current-state brand review per516`reference/brand-review-template.md`.517518The brand-review HTML is the **first surface a human can eyeball** to519verify the extraction before committing to a redesign direction —520misreads that are invisible in JSON (a wrong dominant radius, a521missing system component, a single-page palette bias) are obvious to522the eye while they are still cheap to fix (re-extract is fast;523re-direct + re-prototype is not).524525The template is mandatory. In particular:5265271. Run the **Tensions detectors** listed in528 `reference/brand-review-template.md` § Detectors. Each rule is529 mechanical; emit a tension card whenever the trigger condition530 matches. The review may ship with zero tensions if the data is531 too thin to evaluate, but the detectors must always be run.5322. Render in the brand's **own captured colors and fonts**, not a533 stardust shell.5343. Embed all CSS; do not load external JavaScript or fonts unless535 the live site already does.5364. Cite the source artifact for every section (e.g.537 `_brand-extraction.json § type` under Typography).538539If the data for a section is missing, **omit the section** — do not540fabricate placeholders. The coverage callout at the top reflects what541is missing.542543### Phase 6 — Update state and report544545After all Phase 2-5 writes succeed:5465471. Update `stardust/state.json` (schema in548 `skills/stardust/reference/state-machine.md`):549 - `site.originUrl`, `site.extractedAt`, `site.pageCap`,550 `site.totalDiscovered`, `site.crawled`551 - `pages[]` — one entry per crawled page with `status: "extracted"`,552 filled `currentStatePath`, empty `prototypePath` and `migratedPath`553 - `designSource` — only when `--design-source` was used:554 `{ url, capturedAt, path }` per § Cross-site brand sources5552. Print a one-screen summary:556 ```557 Extracted https://example.com (5/38 pages, sitemap.xml)558559 stardust/current/560 PRODUCT.md, DESIGN.md, DESIGN.json, brand-review.html,561 pages/ (5), assets/logo.svg, _brand-extraction.json, _crawl-log.json562563 Per-page evidence:564 slug live waitMode waitMs status media(img/bg)565 / yes medium 2380 200 38/6566 /pricing yes medium 1940 200 9/0 ⚠ low-media567 /contact yes domcontentloaded(fb) 8000 200 3/0568 ...569570 Wait summary: 4 resolved at medium (avg 2.4s), 1 fallback (timed out at 8s)571 → /contact may be under-captured; consider --refresh572 Media summary: 1 page flagged low-media (/pricing) — see media-coverage check573 Vision check: 4 ok, 1 recaptured (/pricing) — _crawl-log.json#visionCheck574575 Open stardust/current/brand-review.html to verify the extraction576 before running $stardust direct.577578 Coverage note: extracted 5 of 38 discovered pages; 5 pages579 covering distinct templates is usually sufficient for the580 cross-page brand surface. Widen with --cap <N> or --pages.581582 Next: $stardust direct (resolve a redesign direction)583 ```584585 The **per-page evidence table** is mandatory. The `live` column586 is `yes` when `_provenance.renderedBy === "playwright"` AND587 `waitMs > 0`, else `no`. A `no` row means the page record was588 not produced by a live Playwright render — the visible column589 is the defense-in-depth signal for the failure mode the590 write-time guard exists to prevent (2026-04-30 e-commerce run). A591 maintainer scanning the summary should see `yes` on every row.592593 Compute the wait summary by grouping each page's `_provenance.waitMode`594 and averaging `waitMs`. List slugs whose `waitMode` ends in595 `(fallback)` (rendered as `(fb)` in the table for width) as596 candidates for `--refresh`.597598 The **`media(img/bg)` column** is the analogous defense-in-depth599 signal for imagery. It prints `<count of media.imgs> / <count of600 media.cssBackgrounds>`. Flag a row `⚠ low-media` when the page601 reads as brand/marketing (register `brand`, or a landing/solution/602 product template) yet has `cssBackgrounds: []` **and** few large603 rasters (no `media.imgs` entry with intrinsic width ≥ 600). That604 combination is the signature of a silently-failed background /605 lazy-media walk — a capture pass that specs the full background606 walk (`playwright-recipe.md` § Capture list 11) yet silently607 produces nothing still ships an image-less capture (2026-06-26608 a SaaS site: `cssBackgrounds: []` on every page, all product609 imagery lost). A flagged row is the cue to re-run that page with610 `--refresh` (and, if it persists, to fall back to headed Chrome per611 § Bot-management fallback). A maintainer scanning the summary should612 treat a `brand`-register site with all-zero `bg` counts as suspect,613 not as "this site uses no background images."614615## Cross-site brand sources616617Two flags widen extraction beyond the primary origin — **read618`reference/cross-site-sources.md` in full whenever either flag is619present**. Both are opt-in; without them this section is inert.620621Core contract (merge rules and capture shapes in the reference):622623- `--brand-source <url>` (repeatable) — a **same-brand** sibling624 gets a shallow capture (home + ≤2 nav-linked pages, full recipe +625 provenance) under `stardust/current/brand-sources/<host>/`.626 Evidence, not inventory: never enters `state.json.pages[]`.627 Its palette/type/motif/voice evidence aggregates into628 `_brand-extraction.json` with per-origin attribution629 (`origins[]`); conflicts resolve toward the primary; widened630 traits are attributed, never invented. Excluded from system631 components, `voiceTable`, and cross-promo detection.632- `--design-source <url>` — a design donor is captured to633 `stardust/canon-source/` (same shapes as `current/`, default cap),634 its descriptive DESIGN.md/json derived per Phase 4 rules, and635 `state.json.designSource = { url, capturedAt, path:636 "stardust/canon-source/" }` stamped. `direct` pins the donor637 system as the target (its § Mode A); donor evidence never638 aggregates into the primary `_brand-extraction.json`.639640## Sibling-site discovery641642After the primary crawl (Phases 2–2.5), harvest candidate same-brand643origins from evidence **already captured** — no extra navigation:644645- footer / nav links out to other properties646- "our brands" / "our companies" pages647- `hreflang` alternates on other domains648- subdomain families (`shop.`, `careers.`, country subdomains)649- `og:site_name` matches across captured pages650651List candidates with confidence + the evidence line in the crawl652report and in `_crawl-log.json#siblingCandidates[]`:653654```json655{ "origin": "https://example.co.uk", "confidence": "high", "evidence": "hreflang alternate on 4/5 pages", "decision": "included" }656```657658- **Interactive:** propose — *"found 3 candidate sibling properties —659 include as `--brand-source`?"* — and proceed on the answer.660- **Hands-off** (`state.json.handsOff` is true): auto-include up to661 **2 high-confidence** candidates as brand-sources; record the662 decision in `siblingCandidates[].decision`.663664Discovery is capped and cheap: harvesting reads captured evidence665only, and each included sibling gets the shallow brand-source666capture (≤ 3 pages). It must never balloon the crawl.667668## Outputs669670| Path | Purpose |671|---------------------------------------------|-----------------------------------------------------|672| `stardust/current/PRODUCT.md` | Descriptive strategy of the existing site (impeccable format) |673| `stardust/current/DESIGN.md` | Descriptive visual system (Stitch format) |674| `stardust/current/DESIGN.json` | Sidecar with extensions for motifs, voice, components |675| `stardust/current/brand-review.html` | Self-contained visual review of the extraction (first eyeball-able artifact) |676| `stardust/current/pages/<slug>.json` | Per-page parsed structure + content |677| `stardust/current/pages/<slug>.html` | Settled rendered DOM (crawler sidecar; parse offline, never re-scrape) |678| `stardust/current/assets/logo.<ext>` | Extracted logo |679| `stardust/current/assets/favicon.<ext>` | Site favicon (first-class asset; prototype head + deploy consume it) |680| `stardust/current/assets/media/` | Extracted media referenced by pages |681| `stardust/current/assets/screenshots/` | Per-page full-page screenshots, script-captured by `crawl.mjs` (Phase 2.5 vision gate + brand-review) |682| `stardust/current/_brand-extraction.json` | Consolidated brand surface (palette, type, motifs, voice, system components) |683| `stardust/current/_crawl-log.json` | Discovery + crawl audit trail (incl. `visionCheck[]`, `siblingCandidates[]`; `dynamicSurface` reach roll-up only with `--dynamics`) |684| `stardust/current/brand-sources/<host>/` | Shallow same-brand captures (only with `--brand-source`) |685| `stardust/canon-source/` | Design-donor capture + descriptive DESIGN.md/json (only with `--design-source`) |686| `stardust/state.json` | Updated with site + per-page status (+ `designSource` stamp) |687688## Concurrency689690Page captures run **concurrently**: the Phase 2 queue is drained by6914–8 parallel browser contexts (`--concurrency`, default 4). Each692worker owns its `BrowserContext`; consent state is re-established per693context (Setup step 3). Media `resolves` / HEAD checks are batched694with `Promise.all`. Brand-surface aggregation (Phase 3) may proceed695incrementally as pages complete, so long as the written696`_brand-extraction.json` reflects every extracted page. The bundled697crawler implements the pool (`crawl.mjs --concurrency <n>`).698699Across processes, per `state-machine.md`: stardust does not lock. Two700concurrent extracts on the same project are last-write-wins. Document701this in the user report; do not engineer around it.702703## Failure modes704705- **Network failure mid-crawl.** Continue, record in `_crawl-log.json`,706 end with a partial state. State.json reflects only successfully707 extracted pages. User can re-run; already-extracted pages are708 skipped unless `--refresh <slug>`.709- **HTTP 4xx/5xx, non-HTML content, soft-404s.** Validated explicitly710 per `reference/playwright-recipe.md` § Response validation. Each711 produces a distinct error class (`HTTPError`, `ContentTypeError`,712 `EmptyPageError`) recorded in `_crawl-log.json#crawl.failures[]`.713 Failed pages do **not** appear in `state.json` as `extracted` —714 they appear only in the failure log. Without this validation a 5xx715 page silently lands as an empty success and propagates wrong data716 to `direct` and `prototype`.717- **Login wall.** Do not attempt to authenticate. If the home page718 redirects to a login screen, capture that one page, mark the rest as719 unreachable, and ask the user how to proceed (provide cookies via720 Playwright config, change the entry URL, or scope to public pages).721- **Bot-management block (Akamai / Cloudflare / F5 / Imperva / CloudFront).**722 When the first navigation returns `ERR_HTTP2_PROTOCOL_ERROR`,723 `ERR_QUIC_PROTOCOL_ERROR`, or hangs through the hard-cap on a724 TLS/H2 fingerprint check, the issue is JA3/H2 fingerprinting on725 bundled-chromium-default headless mode — not auth, not network.726 `crawl.mjs` runs the fallback itself, window-free: real Chrome727 headless (`channel: 'chrome'`, its own identity, no launch728 flags) first, bundled Chromium only when Chrome is missing, one729 real-Chrome relaunch with the challenge-solve window on an730 entry-page block, and a wall — an edge 403 or a 200 verification731 interstitial — is named within seconds in `_crawl-log.json`732 (`discovery.botBlock`, per-page `BotChallenge` failures), never733 captured as content (`node crawl.mjs --help`; a hand-rolled recipe734 follows `reference/playwright-recipe.md` § Bot-management735 fallback). Note for asset harvest: a page-level bot wall usually736 does NOT gate assets — media/CSS/font URLs commonly return 200 to737 a plain browser-UA curl even while every page navigation is738 challenged (Cloudflare challenge, field-confirmed). Probe one739 asset with curl BEFORE reaching for in-page-fetch machinery; the740 in-page harvest is the fallback, not the default.741- **JavaScript-only content.** Playwright already handles this. If742 the configured wait condition never fires within the mode's hard743 cap (`reference/playwright-recipe.md` § Wait modes), fall back to744 `domcontentloaded` and capture what is rendered. Record the745 fallback in the per-page `_provenance.waitMode` and surface in the746 wait-summary line of the final report.747- **Synthesis attempt (forbidden).** When the agent (or a delegated748 sub-agent) cannot run a real Playwright render for a page —749 whether due to time pressure, token budget, or a tool/networ750751…(truncated)