nukadoko under a verification skill
A verification skill lets an agent check its own work against the real app. It has three parts, and nukadoko is one of them:
- The control CLI drives one long-lived instance of the app: doctor, navigate, snapshot, screenshot, trace, flip a flag. It is for exploring and debugging, and it keeps state between commands.
- The feature map is the project's memory of what the app does, how a user reaches each feature, and what goes wrong there. Much of it is not pass or fail (a tab hidden behind an entitlement, a focus trap), so it stays prose.
- The proof is a nukadoko scenario for each claim that can pass or fail, and the run that is its evidence, bound to a commit.
Write and run scenarios by the nukadoko skill (nukadoko/ beside this
one, under nuka skill path). This file says where scenarios attach to
the map, how the control CLI and the steps share code, and when to reach
for a scenario at all.
1. Where an expected value comes from
The map is written by walking the app, and that is right for the map. A Then is different: written from what the program printed, it grades the implementation against itself.
- Take every expected value in a Then from a promise the project already made: the README, the docs, help text, wording a design document fixes, or the report being reproduced.
- A value no source states goes in the map's gotchas as unsettled, and becomes a question for a person. It never becomes a Then.
- The map's own "what you see" lines may be corrected from what the app does. A Then may not.
- A Then declares claims, and its step record names where each expected
value came from (
sentence:,from:,literal:, orobserve:). Everysentence:andliteral:value traces to one of the sources above. - A failure the program shows underneath a step is judged too: a 5xx
response, an uncaught page error, or an
errorit reports withhandled: falseturns a passed scenario's cellbroke(unexpected-failure). A scenario of an error path a source promises (a maintenance page, a declined card) names the failure it causes with@expects:status=<500-599>,@expects:page-error=<name>, or@expects:error=<class>, from that source, never from what the run saw. - One exception: when the requirement is "behave exactly as before" (a
migration that must keep every behavior, bugs included), the earlier
behavior is the source. Say so in the
Feature:description, so a reader knows why a Then was written from output.
2. Linking the map to scenarios
Keep the map's own sections. End each sub-feature line with the scenario
that proves it, or with no scenario:
- `create-save` persists a title and body. Proof: `features/notes.feature:12`
- `create-cancel` discards an unfinished draft. Proof: no scenario
A link is a <file>:<line> that nuka run accepts, so a reader can run
the proof the line names. no scenario is an honest gap: those
sub-features are the ones a person or an agent still drives by hand.
Scenarios for the product's core path live in featuresDir and run on
every commit. A scenario written for one change stays in a directory named
in additionalFeatureDirs until someone decides it is core.
3. The control CLI and the steps
Write each action and each read once, as a function in a library the steps
already import (for example features/steps/lib/). A step declares its
contract and calls one function; the control CLI calls the same function.
A Then's observe calls a read, and its claims judge what the read
returned; the CLI prints what the same read returns.
For what a page shows, the read returns the locator (or the page), and the
Then claims fits-aria-snapshot on it against a docstring that names roles
and names, so nukadoko takes the ARIA snapshot; the CLI's snapshot command
prints the same locator's ariaSnapshot(). That print is output, so it
explores the page and never becomes the expected tree (section 1).
Keep the library under the features tree and let the CLI import it: steps
that import from the agent's skill directory make the acceptance suite
depend on it.
How a step reaches the CLI's instance depends on its handle:
- The handle is a value (a project directory, a URL with its own data,
a record id). Pass it in
--args, andnuka do <step> --args '<json>'runs one step against the state the CLI built;--usechains the next step from that record. The CLI needs no table of commands that mirrors the steps: one pass-through command (cli <instance> -- <argv>) is enough. - The handle is live (a browser with open tabs and half-typed input).
nuka dostarts clean on every call, so it cannot reach it. The CLI owns the browser: start it with a remote debugging port and reconnect over CDP for each command. Name the CLI's commands after the steps that call the same functions, and prove per-connection settings (scripts turned off) with a scenario, since they do not survive a reconnect.
Either way, run whole scenarios against the CLI's instance: read url
in nukadoko.program.ts from an environment variable whose name no
production tool reads, and have the CLI's scenario command set it and
run nuka run --json. A program each run may start fresh goes in serve
there instead, with its state under {dir}.
A library function that starts a program calls runCommand from
nukadoko. Inside a step the process lands on the step record
(observed.processes); called from the CLI it runs the same way,
unrecorded. Inside a step on POSIX it also runs in a process group and a
session of its own, so a tool that prompts on the terminal has none there,
and the step's events are whole only once nothing of its group still runs.
A program the library starts through node:child_process
instead is only named in the step's observed.unmeasured, by its
program and argument count, with nothing of what it printed or changed. runCommand records the full argv, and
only values loaded from env files or named in redact are redacted, so
pass a token through env, never as an argument. The record names every
variable in env but shows a value only when it is an absolute path that
exists, a decimal integer, or true or false. When the run has a
world, the process also gets TRACEPARENT, the step's trace context
(its entry's traceparent), unless the step set its own: a tool that
copies it into the traceparent header of its requests makes the
program's log lines and execution events for them the step's.
4. Doctor and readiness
serve.ready in nukadoko.program.ts treats a server as ready once one
path answers 2xx, and nothing more. Usable is the control CLI's doctor: the
port is ours, the build is the expected one, the data is seeded. For a
command-line tool, usable is the tool's own startup check (--version
exits 0 on this runtime) and a checkout that has not changed since the
instance was built. Call the same check from the environment's version
probe, so a run against an instance that is up but not usable fails before
its first step, with the reason.
5. Evidence that outlives cleanup
Retention removes old run directories (the newest retention.runs stay),
and nuka clean removes all of them. Evidence for a pull request leaves
the run directory:
npx nuka report "$(npx nuka run --json | jq -r .run_dir)" --format markdown --out <evidence dir>/summary.md
run_dir in run.json is the run directory; environments[].base_url
names the instance the run judged, so the evidence says which one it was.
The summary's seen links (such as steps/<id>/judged/page.png, a Then's
page right after the reading it judged) are relative to the run directory:
copy each file they name into the evidence directory at the same path, or
they lead nowhere there.
or reaches the pull request through CI's run summary (docs/ci.md in the
package). The control CLI's own evidence goes beside it: screenshots and
accessibility snapshots for a web app; for a command-line tool, a
transcript of argv, exit code, stdout, and stderr, and a diff of the files
each command wrote. A step that runs the tool through runCommand
gets that diff measured: each process lists the files added, changed,
and removed under its working directory (observed.processes[].files),
and the step's observed.files lists what changed in its scratch
directories, with the content. A scenario that claims what the tool wrote
("writes nothing", "changed no file under src/") reads that list through a
claim step's files (skills/nukadoko/references/parts.md), never
through a list the step returns. Confirm the evidence exists after the verification skill's cleanup
has run.
6. Reproducing a report
- Restate the report as requirements
(
skills/nukadoko/references/from-prose.md). An unstated slot is a question back to the reporter, not a guess. - Write the scenario from those requirements, before any fix.
- Run it on the base branch. Red reproduces the report; for a report of
a server error or a page that throws, red can be an
unexpected-failureover claims that all held. Green means the scenario does not say what the report says, or the report is about something else: ask. Do not edit the Then until it goes red. - Fix, run again, and open the pull request with
nuka run --base <base>: the new criterion runs against the base's code too, and its row isprovenwhen its Then fails there in each of--proof-runsruns. When the program writes execution events, the summary's "Changed paths" says underbase → headwhat the program did differently at head, step by step, and lists first each unchanged criterion that did something else, which no sentence asks for: read those before calling the fix contained. Before you call it fixed,nuka verdictsays whether that run judged the tree you have and left nothing red (skills/nukadoko/references/done.md). - Link the scenario from the map's sub-feature.
When the scenario and the program file live in the product's repository,
nuka run --base <the commit before the fix> shows red before and green
after in one run: the base world builds the base's code from its program
file. When the scenario lives outside the product's repository, --base
compares the scenario project's commits, not the product's, so showing
both takes two checkouts: put the commit before the fix and the fix commit
in separate worktrees and run the same scenario against each. The run's
commit is then the scenario project's; bind the evidence to the product
by returning the product's git rev-parse HEAD from the environment's
version probe, which the summary shows as the target version.
When the fix sits behind a switch the program reads (a flag, a setting),
one run shows both sides instead: declare the switch in conditions on
two environments of nukadoko.program.ts and tag the scenario
@with:<switch>, so the side without the fix is a control. The verdict
table reads proven when the Then breaks there at the same line in each
of --proof-runs runs, and held when it passes there, which says what
green on the base branch says: the scenario does not say what the report
says (skills/nukadoko/references/verdicts.md). When the program writes
flag execution events, run.json's flags names the side each cell's
steps got: a control listed only under the side with the fix never ran the
side without it. When the two environments' conditions differ in that one
switch alone, run.json's paths and the summary's "Changed paths" say what
the program did differently, step by step, on the side with the fix.
7. Prototypes and plan steps
- Before two or three prototypes are built, write the scenarios they must all pass, and run the same scenarios against each. The run decides which meet the criteria; the control CLI's screenshots and traces compare the rest. Criteria written per prototype after the fact measure each one with its own ruler.
- A plan whose steps each end in a check can name a scenario as the check. The step is done when its scenario is green on the commit that lands it.
8. Keeping the map honest
nuka check: a step or a binding broke, a line needs a state no earlier line leaves (state-missing), and, with no target, notes on steps and fixtures that are rotting.nuka run: a sub-feature with a scenario no longer does what the map says. Red is map drift or a product regression; decide which before editing anything, and never edit a Then to match the product. Aheldrow whose control passed is a scenario that no longer tells its switch apart.nuka run --order shuffled: a scenario that passes only on what an earlier scenario left in the instance they share. A red one'sdo:line replays the scenarios that ran before it in that order. Red again there is an order dependence, fixed by a scenario that stands alone, never by a fixed order.nuka run --base <ref>: which criteria changed since the map was last checked, and, when the steps changed, whether an unchanged one still means what it meant (amovedrow says it does not).nuka runwithlistsin the program file: the routes the program has that no scenario requested ("Routes" in the summary,routesin run.json;skills/nukadoko/references/verdicts.md). A route no scenario requested is a place the map may be missing; a requested one is not yet a checked one.nuka runwith atableslist in the program file: a count of writes that names no table counts the tables that hold state, and its line names the bookkeeping and infrastructure writes it left out; the noteundeclared-writenames a step declaredmutates: falsewhose trace the program says wrote state (skills/nukadoko/references/diagnosing.md). Either is a question for the person, not a Then to edit.
What remains for a pass that drives the app by hand is the sub-features
marked no scenario, the routes no scenario requested, and features
missing from the map.