# Nukadoko Verification

> Use when a project has, or is getting, a verification skill (a control CLI that drives the running app, plus a feature map of what the app does and how a user reaches it) and its claims need evidence that can be run again; when a bug report has to be reproduced before it is fixed; or when prototypes or the steps of a plan need one yardstick fixed before they are built. Adds nukadoko scenarios as the proof layer under the map. It does not replace the control CLI or the map.

- Skill: `meganemura/nukadoko-verification` (Agent Skill)
- Install (CLI): `npx get-skill add meganemura/nukadoko-verification`
- Raw SKILL.md: https://api.skillmd.com/api/skills/meganemura/nukadoko-verification/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- License: MIT
- Author: meganemura (https://skillmd.com/u/meganemura)
- Updated: 2026-11-15
- Page: https://skillmd.com/skills/meganemura/nukadoko-verification

---


# nukadoko under a verification skill

A verification skill lets an agent check its own work against the real
app. It has three parts, and nukadoko is one of them:

- **The control CLI** drives one long-lived instance of the app: doctor,
  navigate, snapshot, screenshot, trace, flip a flag. It is for exploring
  and debugging, and it keeps state between commands.
- **The feature map** is the project's memory of what the app does, how a
  user reaches each feature, and what goes wrong there. Much of it is not
  pass or fail (a tab hidden behind an entitlement, a focus trap), so it
  stays prose.
- **The proof** is a nukadoko scenario for each claim that can pass or
  fail, and the run that is its evidence, bound to a commit.

Write and run scenarios by the nukadoko skill (`nukadoko/` beside this
one, under `nuka skill path`). This file says where scenarios attach to
the map, how the control CLI and the steps share code, and when to reach
for a scenario at all.

## 1. Where an expected value comes from

The map is written by walking the app, and that is right for the map. A
Then is different: written from what the program printed, it grades the
implementation against itself.

- Take every expected value in a Then from a promise the project already
  made: the README, the docs, help text, wording a design document fixes,
  or the report being reproduced.
- A value no source states goes in the map's gotchas as unsettled, and
  becomes a question for a person. It never becomes a Then.
- The map's own "what you see" lines may be corrected from what the app
  does. A Then may not.
- A Then declares claims, and its step record names where each expected
  value came from (`sentence:`, `from:`, `literal:`, or `observe:`).
  Every `sentence:` and `literal:` value traces to one of the sources
  above.
- A failure the program shows underneath a step is judged too: a 5xx
  response, an uncaught page error, or an `error` it reports with
  `handled: false` turns a passed scenario's cell `broke`
  (`unexpected-failure`). A scenario of an error path a source promises
  (a maintenance page, a declined card) names the failure it causes with
  `@expects:status=<500-599>`, `@expects:page-error=<name>`, or
  `@expects:error=<class>`, from that source, never from what the run saw.
- One exception: when the requirement is "behave exactly as before" (a
  migration that must keep every behavior, bugs included), the earlier
  behavior is the source. Say so in the `Feature:` description, so a
  reader knows why a Then was written from output.

## 2. Linking the map to scenarios

Keep the map's own sections. End each sub-feature line with the scenario
that proves it, or with `no scenario`:

```markdown
- `create-save` persists a title and body. Proof: `features/notes.feature:12`
- `create-cancel` discards an unfinished draft. Proof: no scenario
```

A link is a `<file>:<line>` that `nuka run` accepts, so a reader can run
the proof the line names. `no scenario` is an honest gap: those
sub-features are the ones a person or an agent still drives by hand.

Scenarios for the product's core path live in `featuresDir` and run on
every commit. A scenario written for one change stays in a directory named
in `additionalFeatureDirs` until someone decides it is core.

## 3. The control CLI and the steps

Write each action and each read once, as a function in a library the steps
already import (for example `features/steps/lib/`). A step declares its
contract and calls one function; the control CLI calls the same function.
A Then's `observe` calls a read, and its claims judge what the read
returned; the CLI prints what the same read returns.
For what a page shows, the read returns the locator (or the page), and the
Then claims `fits-aria-snapshot` on it against a docstring that names roles
and names, so nukadoko takes the ARIA snapshot; the CLI's `snapshot` command
prints the same locator's `ariaSnapshot()`. That print is output, so it
explores the page and never becomes the expected tree (section 1).
Keep the library under the features tree and let the CLI import it: steps
that import from the agent's skill directory make the acceptance suite
depend on it.

How a step reaches the CLI's instance depends on its handle:

- **The handle is a value** (a project directory, a URL with its own data,
  a record id). Pass it in `--args`, and `nuka do <step> --args '<json>'`
  runs one step against the state the CLI built; `--use` chains the next
  step from that record. The CLI needs no table of commands that mirrors
  the steps: one pass-through command (`cli <instance> -- <argv>`) is
  enough.
- **The handle is live** (a browser with open tabs and half-typed input).
  `nuka do` starts clean on every call, so it cannot reach it. The CLI
  owns the browser: start it with a remote debugging port and reconnect
  over CDP for each command. Name the CLI's commands after the steps that
  call the same functions, and prove per-connection settings (scripts
  turned off) with a scenario, since they do not survive a reconnect.

Either way, run whole scenarios against the CLI's instance: read `url`
in `nukadoko.program.ts` from an environment variable whose name no
production tool reads, and have the CLI's `scenario` command set it and
run `nuka run --json`. A program each run may start fresh goes in `serve`
there instead, with its state under `{dir}`.

A library function that starts a program calls `runCommand` from
`nukadoko`. Inside a step the process lands on the step record
(`observed.processes`); called from the CLI it runs the same way,
unrecorded. Inside a step on POSIX it also runs in a process group and a
session of its own, so a tool that prompts on the terminal has none there,
and the step's events are whole only once nothing of its group still runs.
A program the library starts through `node:child_process`
instead is only named in the step's `observed.unmeasured`, by its
program and argument count, with nothing of what it printed or changed. `runCommand` records the full argv, and
only values loaded from env files or named in `redact` are redacted, so
pass a token through `env`, never as an argument. The record names every
variable in `env` but shows a value only when it is an absolute path that
exists, a decimal integer, or `true` or `false`. When the run has a
world, the process also gets `TRACEPARENT`, the step's trace context
(its entry's `traceparent`), unless the step set its own: a tool that
copies it into the `traceparent` header of its requests makes the
program's log lines and execution events for them the step's.

## 4. Doctor and readiness

`serve.ready` in `nukadoko.program.ts` treats a server as ready once one
path answers 2xx, and nothing more. Usable is the control CLI's `doctor`: the
port is ours, the build is the expected one, the data is seeded. For a
command-line tool, usable is the tool's own startup check (`--version`
exits 0 on this runtime) and a checkout that has not changed since the
instance was built. Call the same check from the environment's `version`
probe, so a run against an instance that is up but not usable fails before
its first step, with the reason.

## 5. Evidence that outlives cleanup

Retention removes old run directories (the newest `retention.runs` stay),
and `nuka clean` removes all of them. Evidence for a pull request leaves
the run directory:

```sh
npx nuka report "$(npx nuka run --json | jq -r .run_dir)" --format markdown --out <evidence dir>/summary.md
```

`run_dir` in run.json is the run directory; `environments[].base_url`
names the instance the run judged, so the evidence says which one it was.
The summary's `seen` links (such as `steps/<id>/judged/page.png`, a Then's
page right after the reading it judged) are relative to the run directory:
copy each file they name into the evidence directory at the same path, or
they lead nowhere there.

or reaches the pull request through CI's run summary (`docs/ci.md` in the
package). The control CLI's own evidence goes beside it: screenshots and
accessibility snapshots for a web app; for a command-line tool, a
transcript of argv, exit code, stdout, and stderr, and a diff of the files
each command wrote. A step that runs the tool through `runCommand`
gets that diff measured: each process lists the files added, changed,
and removed under its working directory (`observed.processes[].files`),
and the step's `observed.files` lists what changed in its scratch
directories, with the content. A scenario that claims what the tool wrote
("writes nothing", "changed no file under src/") reads that list through a
claim step's `files` (`skills/nukadoko/references/parts.md`), never
through a list the step returns. Confirm the evidence exists after the verification skill's cleanup
has run.

## 6. Reproducing a report

1. Restate the report as requirements
   (`skills/nukadoko/references/from-prose.md`). An unstated slot is a
   question back to the reporter, not a guess.
2. Write the scenario from those requirements, before any fix.
3. Run it on the base branch. Red reproduces the report; for a report of
   a server error or a page that throws, red can be an
   `unexpected-failure` over claims that all held. Green means the
   scenario does not say what the report says, or the report is about
   something else: ask. Do not edit the Then until it goes red.
4. Fix, run again, and open the pull request with
   `nuka run --base <base>`: the new criterion runs against the base's
   code too, and its row is `proven` when its Then fails there in each of
   `--proof-runs` runs. When the program writes execution events, the
   summary's "Changed paths" says under `base → head` what the program
   did differently at head, step by step, and lists first each unchanged
   criterion that did something else, which no sentence asks for: read
   those before calling the fix contained. Before you call it fixed,
   `nuka verdict` says whether that run judged the tree you have and left
   nothing red (`skills/nukadoko/references/done.md`).
5. Link the scenario from the map's sub-feature.

When the scenario and the program file live in the product's repository,
`nuka run --base <the commit before the fix>` shows red before and green
after in one run: the base world builds the base's code from its program
file. When the scenario lives outside the product's repository, `--base`
compares the scenario project's commits, not the product's, so showing
both takes two checkouts: put the commit before the fix and the fix commit
in separate worktrees and run the same scenario against each. The run's
`commit` is then the scenario project's; bind the evidence to the product
by returning the product's `git rev-parse HEAD` from the environment's
`version` probe, which the summary shows as the target version.

When the fix sits behind a switch the program reads (a flag, a setting),
one run shows both sides instead: declare the switch in `conditions` on
two environments of `nukadoko.program.ts` and tag the scenario
`@with:<switch>`, so the side without the fix is a control. The verdict
table reads `proven` when the Then breaks there at the same line in each
of `--proof-runs` runs, and `held` when it passes there, which says what
green on the base branch says: the scenario does not say what the report
says (`skills/nukadoko/references/verdicts.md`). When the program writes
`flag` execution events, run.json's `flags` names the side each cell's
steps got: a control listed only under the side with the fix never ran the
side without it. When the two environments' `conditions` differ in that one
switch alone, run.json's `paths` and the summary's "Changed paths" say what
the program did differently, step by step, on the side with the fix.

## 7. Prototypes and plan steps

- Before two or three prototypes are built, write the scenarios they must
  all pass, and run the same scenarios against each. The run decides which
  meet the criteria; the control CLI's screenshots and traces compare the
  rest. Criteria written per prototype after the fact measure each one
  with its own ruler.
- A plan whose steps each end in a check can name a scenario as the check.
  The step is done when its scenario is green on the commit that lands it.

## 8. Keeping the map honest

- `nuka check`: a step or a binding broke, a line needs a state no
  earlier line leaves (`state-missing`), and, with no target, notes on
  steps and fixtures that are rotting.
- `nuka run`: a sub-feature with a scenario no longer does what the map
  says. Red is map drift or a product regression; decide which before
  editing anything, and never edit a Then to match the product. A `held`
  row whose control passed is a scenario that no longer tells its switch
  apart.
- `nuka run --order shuffled`: a scenario that passes only on what an
  earlier scenario left in the instance they share. A red one's `do:` line
  replays the scenarios that ran before it in that order. Red again there
  is an order dependence, fixed by a scenario that stands alone, never by a
  fixed order.
- `nuka run --base <ref>`: which criteria changed since the map was last
  checked, and, when the steps changed, whether an unchanged one still
  means what it meant (a `moved` row says it does not).
- `nuka run` with `lists` in the program file: the routes the program
  has that no scenario requested ("Routes" in the summary, `routes` in
  run.json; `skills/nukadoko/references/verdicts.md`). A route no
  scenario requested is a place the map may be missing; a requested one
  is not yet a checked one.
- `nuka run` with a `tables` list in the program file: a count of
  writes that names no table counts the tables that hold state, and its
  line names the bookkeeping and infrastructure writes it left out; the
  note `undeclared-write` names a step declared `mutates: false` whose
  trace the program says wrote state
  (`skills/nukadoko/references/diagnosing.md`). Either is a question for
  the person, not a Then to edit.

What remains for a pass that drives the app by hand is the sub-features
marked `no scenario`, the routes no scenario requested, and features
missing from the map.

