Test The Docs

Test the docs

Runs procedural docs content inside disposable containers, not on the host shell and not against production. Produces a verification report for the PR body / self-review note.

For lint, markdown rebuilds, example-app triage, and PR review, use review-the-docs. For Frame/Shape and cross-repo product lookup, use pm-the-docs.

When to invoke

• After Draft, before or during Self-review (checklist Stage 4).
• Standalone: "test the docs", "fact-check these snippets", "run the examples".
• Content or tutorial PRs that add or change procedural fenced blocks.

Not for: generated reference pages, docs-app architecture questions, or hosted/production projects.

Core rules

1. Never run against production. Local stack or temp dir only.
2. Never run MDX fences on the host shell. Use the Compose sandbox — see reference/sandbox-setup.md and sandbox/run.sh.
3. Proportional: Tier A (one end-to-end path) is required; Tier B spot-checks new/changed procedural blocks, not every fence on every page.
4. Product bugs found while testing get linked or filed separately; fix docs only when the docs are wrong.

Phases

1. Scope

From explicit MDX paths, or:

git diff --name-only master...HEAD -- 'apps/docs/content/**'

Skip generated reference output under features/docs/generated/.

2. Extract

List runnable artifacts from changed MDX:

• Fenced blocks: bash, sh, sql, javascript, typescript, tsx, jsx
• $CodeSample paths → treat as example-app (build under examples/)
• Skip: mermaid, incomplete illustrative fragments, partial-only includes

3. Classify

Assign each artifact a class per reference/snippet-classes.md:

4. Sandbox setup

Follow reference/sandbox-setup.md and drive lifecycle with sandbox/run.sh:

1. Refuse if the host is running as root.
2. Require docker + docker info + docker compose on the host for any in-container run.
3. Gate profiles per artifact class:
   • runnable-local / runnable-with-setup that need DB/API: ./sandbox/run.sh up-stack (DinD + runner → supabase init / supabase start in /work).
   • CLI-only blocks with no DB: still use a runner profile so fences stay off-host; skip supabase start when unused.
   • example-app: TTD_EXAMPLE_DIR=<repo>/examples/<app> ./sandbox/run.sh up-examples (Node in runner; no DinD). Do not defer solely because the host lacks a global Supabase CLI.
4. Always ./sandbox/run.sh down when finished (cleanup trap on the host session).
5. Capture connection URLs only inside the runner; never paste credential fields into notes or logs.

If a required prerequisite for that artifact is unavailable, mark that artifact deferred with the specific reason — never silent skip, and do not defer unrelated classes.

5. Execute

• Tier A: one copy-pasteable end-to-end path from the page.
• Tier B: each new/changed block classified runnable-* or example-app (build and record the result).
• Run every fence via ./sandbox/run.sh exec or exec-timeout (never a bare host shell).
• Bound every artifact: default 60s for shell / SQL / JS / TypeScript / tsx / jsx; allow longer for example-app install/build (e.g. 5m). On timeout, kill the process group inside the runner, then record fail or deferred with reason.
• curl / wget only to filtered stack URLs (or page-documented local endpoints). npm / npx / node only for mounted example-app builds.
• Capture exit code, stdout/stderr (redact secrets), and observed vs expected behavior.

6. Report

Write a verification report per reference/verification-report.md for the PR body / self-review note.

Related skills

• write-the-docs — Draft; hands off here before PR
• review-the-docs — lint/build/classify; consumes verification report
• pm-the-docs — Frame/Shape; universe for cross-repo product lookup