# Snippet Verify

> Execute the code in your documentation and report which blocks actually run. Extracts every fenced snippet from a docs directory, README, MDX tree, or public docs URL, classifies each as runnable, credential-blocked, or illustrative, runs the runnable ones in a throwaway workspace against the versions you actually publish, and reports a pass rate with the real error for every failure. Use when asked whether the documentation still works, to verify code samples or quickstart steps, to check examples against a new release, before a launch, or when developers report that the docs do not run.

- Skill: `agentrel/snippet-verify` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add agentrel/snippet-verify`
- Raw SKILL.md: https://api.skillmd.com/api/skills/agentrel/snippet-verify/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: agentrel (https://skillmd.com/u/agentrel)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/agentrel/snippet-verify

---


# Snippet verify

Every other check in this plugin produces a judgement. This one produces a fact: the number
of documented code samples that execute, and the exact error for each one that does not.

That number is not arguable, which is what makes it useful. A broken quickstart is the most
expensive defect a developer product can ship, and it is invisible to code review because
the code that broke lives in prose.

## Applies to

| | |
|---|---|
| **Project types** | Anything documenting code — SDKs, APIs, CLIs, frameworks, libraries, developer platforms |
| **Stage** | Any product with published examples. Highest value right after a major release, before a launch, and on a schedule thereafter — samples rot silently between releases |
| **Needs** | Documentation containing fenced code blocks, plus a runtime for the languages used (`node`, `python`, `go`, `curl`) |
| **Skip if** | The docs are conceptual only, with no executable samples. It will report that and stop rather than manufacture findings |

Most valuable on the quickstart. A failing sample on page forty costs one reader; a failing
sample on the first page costs every reader.

## How to use

```
/agentrel:snippet-verify                          verify this repository's docs
/agentrel:snippet-verify ./docs/quickstart.mdx    verify one page
/agentrel:snippet-verify https://docs.acme.com    verify a public docs site
/agentrel:snippet-verify --lang python            verify one language
```

Takes several minutes — dependency installation dominates. Produces
`snippet-verify-report.md` plus a printed pass rate.

**Nothing in the project is modified.** All execution happens in a throwaway workspace
outside the project tree, and the skill never overwrites a file in the repository being
checked. The one file it writes is the report.

## 1. Inventory the snippets

Collect every fenced block with its source location. The location is what makes a failure
actionable — a pass rate with no `file:line` is a statistic, not a bug report.

| Source | Where to look |
|--------|---------------|
| Repository docs | `docs/`, `README.md`, `*.md`, `*.mdx`, `examples/` |
| Framework docs | `docusaurus`, `mint.json`, `fern.config.json`, `.vitepress`, `astro` content collections |
| Docstrings | Python docstrings, JSDoc `@example`, Rust doc tests |
| Public site | Fetch the docs root, follow the quickstart and reference paths |

Record for each block: source path and line, declared language, and any surrounding
instruction that changes how it runs — a preceding "install this first", a filename comment,
a `# in a new terminal`.

Blocks with no language tag are still inventoried. An untagged block is itself a finding:
no syntax highlighting, and no way for a tool or an agent to know how to run it.

## 2. Classify before running

**This is the step that decides whether the report is credible.** Not every fenced block is
meant to execute, and running an illustrative fragment to declare it broken makes the tool a
liar. Read `references/classification.md` for the full rules. In summary:

| Class | Means | Counted in the score |
|-------|-------|----------------------|
| `runnable` | Self-contained, or completable from documented prior steps, with no secret required | Yes |
| `needs-credentials` | Correct code that cannot run without an account, key, or paid resource | No — reported separately |
| `illustrative` | A fragment, a schema, a response payload, pseudocode, output of another command | No |
| `unsafe` | Would delete data, spend money, or execute a remote script | No — never run, always reported |

State the class and the reason for every block. The pass rate is computed over `runnable`
blocks only, and the report says so on the same line as the number. A tool that quietly
counts illustrative fragments as failures is worse than no tool.

## 3. Build a throwaway workspace

One directory per language, outside the project tree, created fresh:

```bash
WORK="$(mktemp -d)"            # never inside the repository being checked
```

Pin the versions the product actually publishes, not `latest` — the point is to verify what
a reader gets today:

- read the published version from `package.json`, `pyproject.toml`, the registry, or the
  install line in the docs themselves
- install into the workspace only, never globally, never into the project
- record every install command and its resolved versions in the report

**Ask before installing.** Dependency installation uses network and disk. Say what will be
installed and how large it is, then wait. If the user declines, verify what runs without
installation and mark the rest `skipped — install declined`.

## 4. Run them

Execute each `runnable` block in its workspace, in documentation order, carrying forward the
state earlier blocks established — a snippet that depends on a variable defined two blocks
up is only valid in sequence, and testing it in isolation invents a failure.

Capture for each: exit status, stdout, stderr, and duration. Record the **first** error line,
verbatim. Paraphrased errors are useless to the person fixing them.

Rules that are not negotiable:

- **No writes outside the workspace.** Not to the repository, not to the home directory, not
  to a global cache the project shares.
- **No real credentials.** Never read `.env`, never use an existing shell token, never prompt
  for a key. A block needing auth is `needs-credentials`, not a failure.
- **No live mutation.** A documented `POST`, `DELETE`, or destructive CLI command against a
  real service is `unsafe` unless the docs name a sandbox host and the snippet uses it.
- **No remote execution.** `curl … | sh`, `iex`, and equivalents are `unsafe`, reported and
  never run, however common they are in install instructions.
- **Timeout everything.** A hung snippet is a finding — record it as a failure with the
  timeout, do not wait.

## 5. Diagnose each failure

A pass rate alone tells a team they have a problem. The cause tells them which line to edit.
Classify every failure:

| Cause | Signature | Typical fix |
|-------|-----------|-------------|
| Stale API | Method or field does not exist | Update the sample to the current signature |
| Missing prerequisite | Import, install, or setup step absent from the page | Add the missing step to the page |
| Version drift | Works on an older release, not the published one | Re-generate the sample, or pin in the docs |
| Wrong order | Depends on a block that appears later | Reorder the page |
| Placeholder | A value the reader is expected to replace, unmarked | Mark it, or use an obviously fake value |
| Broken syntax | Never ran anywhere | Fix or delete the block |

"Placeholder" and "wrong order" are documentation defects, not code defects. Report them as
such — the fix belongs to whoever owns the page.

## 6. Write the report

Write `snippet-verify-report.md` with these sections, in order:

1. **Scope** — what was scanned, which languages, which versions were installed
2. **Pass rate** — over `runnable` blocks, with every other class counted separately
3. **Failures** — each with `file:line`, class, the verbatim first error, and the cause
4. **Not run** — `needs-credentials`, `illustrative`, `unsafe`, `skipped`, each with the reason
5. **Method** — commands, resolved versions, date, and the limits of this run

Then print this summary. Keep the format exact; other tools parse it.

```
SNIPPET VERIFY

  Blocks found            128
  Runnable                 74
  Passed                   61      82%
  Failed                   13

  Not run                  54      illustrative 38 · needs-credentials 14 · unsafe 2

  Worst page   docs/quickstart.mdx      4 of 6 runnable blocks fail
  Top cause    stale API signature      7 failures

Report → ./snippet-verify-report.md
```

## 7. Close honestly

Lead with the quickstart. A product whose reference pages are perfect and whose quickstart
fails is in worse shape than the aggregate suggests, so say that explicitly when it is true.

If every runnable block passes, report it in two lines and stop. A clean run is a real
result, and reporting it plainly is what makes a bad run credible.

## Notes

- **Never overwrite anything.** Execution happens in a fresh temporary workspace; the only
  file written is the report, in the working directory. If a report already exists, write
  alongside it rather than replacing it.
- **The score covers runnable blocks only**, and every statement of the score must carry that
  qualifier. Silent denominators are how these tools lose trust.
- **Public-site runs see less than repository runs.** A fetched page loses the surrounding
  setup instructions that make a block runnable. Prefer the repository, and label remote
  runs as such.
- **This is the one skill here that executes code.** It is opt-in per run, it asks before
  installing, and it refuses the four unsafe categories above without exception.
- **No telemetry.** Results stay on the machine.