Snippet Verify

Snippet verify

Every other check in this plugin produces a judgement. This one produces a fact: the number of documented code samples that execute, and the exact error for each one that does not.

That number is not arguable, which is what makes it useful. A broken quickstart is the most expensive defect a developer product can ship, and it is invisible to code review because the code that broke lives in prose.

Applies to

Most valuable on the quickstart. A failing sample on page forty costs one reader; a failing sample on the first page costs every reader.

How to use

/agentrel:snippet-verify                          verify this repository's docs
/agentrel:snippet-verify ./docs/quickstart.mdx    verify one page
/agentrel:snippet-verify https://docs.acme.com    verify a public docs site
/agentrel:snippet-verify --lang python            verify one language

Takes several minutes — dependency installation dominates. Produces snippet-verify-report.md plus a printed pass rate.

Nothing in the project is modified. All execution happens in a throwaway workspace outside the project tree, and the skill never overwrites a file in the repository being checked. The one file it writes is the report.

1. Inventory the snippets

Collect every fenced block with its source location. The location is what makes a failure actionable — a pass rate with no file:line is a statistic, not a bug report.

Record for each block: source path and line, declared language, and any surrounding instruction that changes how it runs — a preceding "install this first", a filename comment, a # in a new terminal.

Blocks with no language tag are still inventoried. An untagged block is itself a finding: no syntax highlighting, and no way for a tool or an agent to know how to run it.

2. Classify before running

This is the step that decides whether the report is credible. Not every fenced block is meant to execute, and running an illustrative fragment to declare it broken makes the tool a liar. Read references/classification.md for the full rules. In summary:

State the class and the reason for every block. The pass rate is computed over runnable blocks only, and the report says so on the same line as the number. A tool that quietly counts illustrative fragments as failures is worse than no tool.

3. Build a throwaway workspace

One directory per language, outside the project tree, created fresh:

WORK="$(mktemp -d)"            # never inside the repository being checked

Pin the versions the product actually publishes, not latest — the point is to verify what a reader gets today:

• read the published version from package.json, pyproject.toml, the registry, or the install line in the docs themselves
• install into the workspace only, never globally, never into the project
• record every install command and its resolved versions in the report

Ask before installing. Dependency installation uses network and disk. Say what will be installed and how large it is, then wait. If the user declines, verify what runs without installation and mark the rest skipped — install declined.

4. Run them

Execute each runnable block in its workspace, in documentation order, carrying forward the state earlier blocks established — a snippet that depends on a variable defined two blocks up is only valid in sequence, and testing it in isolation invents a failure.

Capture for each: exit status, stdout, stderr, and duration. Record the first error line, verbatim. Paraphrased errors are useless to the person fixing them.

Rules that are not negotiable:

• No writes outside the workspace. Not to the repository, not to the home directory, not to a global cache the project shares.
• No real credentials. Never read .env, never use an existing shell token, never prompt for a key. A block needing auth is needs-credentials, not a failure.
• No live mutation. A documented POST, DELETE, or destructive CLI command against a real service is unsafe unless the docs name a sandbox host and the snippet uses it.
• No remote execution. curl … | sh, iex, and equivalents are unsafe, reported and never run, however common they are in install instructions.
• Timeout everything. A hung snippet is a finding — record it as a failure with the timeout, do not wait.

5. Diagnose each failure

A pass rate alone tells a team they have a problem. The cause tells them which line to edit. Classify every failure:

"Placeholder" and "wrong order" are documentation defects, not code defects. Report them as such — the fix belongs to whoever owns the page.

6. Write the report

Write snippet-verify-report.md with these sections, in order:

1. Scope — what was scanned, which languages, which versions were installed
2. Pass rate — over runnable blocks, with every other class counted separately
3. Failures — each with file:line, class, the verbatim first error, and the cause
4. Not run — needs-credentials, illustrative, unsafe, skipped, each with the reason
5. Method — commands, resolved versions, date, and the limits of this run

Then print this summary. Keep the format exact; other tools parse it.

SNIPPET VERIFY

  Blocks found            128
  Runnable                 74
  Passed                   61      82%
  Failed                   13

  Not run                  54      illustrative 38 · needs-credentials 14 · unsafe 2

  Worst page   docs/quickstart.mdx      4 of 6 runnable blocks fail
  Top cause    stale API signature      7 failures

Report → ./snippet-verify-report.md

7. Close honestly

Lead with the quickstart. A product whose reference pages are perfect and whose quickstart fails is in worse shape than the aggregate suggests, so say that explicitly when it is true.

If every runnable block passes, report it in two lines and stop. A clean run is a real result, and reporting it plainly is what makes a bad run credible.

Notes

• Never overwrite anything. Execution happens in a fresh temporary workspace; the only file written is the report, in the working directory. If a report already exists, write alongside it rather than replacing it.
• The score covers runnable blocks only, and every statement of the score must carry that qualifier. Silent denominators are how these tools lose trust.
• Public-site runs see less than repository runs. A fetched page loses the surrounding setup instructions that make a block runnable. Prefer the repository, and label remote runs as such.
• This is the one skill here that executes code. It is opt-in per run, it asks before installing, and it refuses the four unsafe categories above without exception.
• No telemetry. Results stay on the machine.