# Verification Authoring > Author Duhem Verification Definitions (criteria + checks YAML) that verify your product's features. Use when asked to "write a verification", "spec acceptance criteria", "author checks", "verify this feature", "translate criteria to checks", "write criteria", or "draft a Verification Definition" against a product you verify with Duhem. Enforces Duhem's Holistic Verification Principle (no mocks of the web), the criteria-vs-checks separation (criteria stable, checks derivative), and the mechanical-judgment rule (no LLM-in-the-loop verdicts). - Skill: `onsager-ai/verification-authoring-2` (Agent Skill) - Install (CLI): `npx skillmds@latest add onsager-ai/verification-authoring-2` - Raw SKILL.md: https://api.skillmd.com/api/skills/onsager-ai/verification-authoring-2/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: onsager-ai (https://skillmd.com/u/onsager-ai) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/onsager-ai/verification-authoring-2 --- # verification-authoring Author Verification Definitions for your product — the structured YAML artifact holding a feature's acceptance criteria and the checks that mechanically verify them. This is the central authoring discipline for using Duhem to verify a product from the outside. A Verification Definition exercises the **real, deployed web** — code + prompts + tool wiring + data + runtime — together, and produces a verdict by deterministic evaluation of structured assertions. No LLM is in the verdict loop. **Retrieve, don't recall.** Duhem's action catalog is versioned and evolves; don't guess `with:` keys or output names from memory. Ask the CLI for the version-exact contract as you author: ```bash duhem actions # list the action catalog duhem describe api/call # one action's with: / outputs / example duhem validate # field-check your VD against the catalog duhem mcp # expose describe/actions/validate to your # coding agent over MCP, for AI-authoring ``` Author → `validate` → fix converges to a correct VD without any prior Duhem knowledge. That's retrieval plus verification, not pretraining. ## When to use this skill Use when: - You're writing a Verification Definition for a feature of your product. - Someone asks to "write criteria" or "translate criteria to checks". - A feature needs a mechanically-judged acceptance test that exercises the real system end-to-end. Skip when: - The change is purely internal (build config, refactor with no externally observable behavior change). Those are unit/integration tests in your own toolchain, not Verification Definitions. ## Where the VD lives Verification Definitions live **in your product repo**, co-located with the product under a `.duhem/` suite. Scaffold one with: ```bash duhem init --name .duhem/ # runnable skeleton in .duhem/ ``` (`duhem init` defaults its target to `./verifications//`, so pass the `.duhem/` path explicitly to land it in your co-located suite.) The skeleton is a single passing check against a known-good baseline — mutate it toward your feature. `duhem run` discovers the `.duhem/` manifest by walking up from the current directory (capped at the enclosing `.git`), so `duhem run` from anywhere in the repo finds it; `-f path/to/duhem.yml` overrides discovery. Self-gate the suite in your product's own CI: run `duhem run` on the PR and let the verdict gate merge. ## The two-layer structure Duhem deliberately separates a feature's commitments from the mechanism that verifies them: ``` Criterion → natural language, human-authored, stable across implementations. Checks → structured YAML, AI-translatable, frozen after human review, one-or-many per criterion. ``` **Criteria are stable; checks are derivative.** When the implementation changes, criteria do not. When criteria change, that's a real change to the contract. Write criteria first, then translate each into checks. If a criterion already exists (lifted from a spec / acceptance test), copy it verbatim and translate mechanically — don't re-author it. ## Authoring loop ``` 0. Scaffold: `duhem init --name .duhem/` → a runnable skeleton in `.duhem/` (a single passing baseline check to mutate). Pass the path explicitly — `init`'s default target is `./verifications//`. 1. Lift criteria from the spec / acceptance test / PRD. 2. Validate the criteria are stable, intent-bearing, one-commitment-each. 3. Translate each criterion into one or more checks (steps + assertions). 4. Review the holistic-environment tax — no mocks of the web. 5. Self-validate: every assertion mechanical, every referenced output reachable, every check has a verdict. 6. Save the file (self-identifies via top-level `verification:`). 7. Register it in `.duhem/duhem.yml` if the suite uses a manifest. ``` ### 1. Lift criteria Find the source of intent for the feature — a spec's acceptance items, a PRD, or the "what does done mean" answer a stakeholder gives. Each criterion is **a single coherent commitment, in 1–3 sentences, that a non-technical stakeholder can read and validate.** A feature typically has 2–6 criteria. A criterion expresses *intent*, not *procedure*: > ✅ A user can create a workspace from the dashboard. The new workspace > becomes immediately visible in their workspace list, and the user is > navigated to the workspace's home page. No errors are shown. > > ❌ When the user clicks "Create Workspace", the system POSTs to > `/workspaces` with `{name, owner_id}`, receives a 200 with the > workspace ID, then redirects to `/workspaces/`. The first describes what "done" means. The second describes *how* — and would have to be rewritten any time the implementation changes. That's a check, not a criterion. ### 2. Validate criteria Before translating, sanity-check each criterion: - [ ] One coherent commitment (not "and also and also …"). - [ ] 1–3 sentences. - [ ] Free of implementation language (no endpoint paths, function names, DB tables). - [ ] Free of step-by-step procedure. - [ ] A non-technical stakeholder could read it and say yes/no. - [ ] Stable across plausible implementation changes. If a criterion violates any of these, rewrite it before translating. ### 3. Translate to checks (terse by default) Each criterion gets one or more checks. A check is a sequence of `steps` (named actions) followed by `assertions` (mechanical predicates over named outputs). **A single check should exercise a slice of the holistic web** — UI input, network, API shape, data — not a single component. Decomposing a check into per-component sub-checks loses what makes Duhem Duhem. Author the **terse form** — Duhem infers the ceremony: - **`outputs:` is optional.** Reference any output an action declares directly as `$steps..outputs.` — including nested paths (`$steps.home.outputs.body.data._id`). No `outputs:` block needed just to name a field. Add `outputs:` only to *rename* a field (`outputs: { code: status }`) or bind a *deep extraction* to a short alias (`outputs: { project_id: body.data._id }`) so you write the path once. An identity binding like `outputs: { foo: foo }` is a redundant no-op — `validate` flags it. - **`assertions:` is optional for a judging step.** A `ui/assert-*` (or `api/poll`) step *is* the judgment — it emits `satisfied` and Duhem implicitly asserts `satisfied == true`. An all-assert check needs no `assertions:` block at all. Bind `satisfied` and assert it yourself only for manual control (e.g. a disjunction across steps). A check with neither `assertions:` nor a judging step is rejected at validate time. Retrieve each action's real `with:` fields and outputs with `duhem describe ` before writing the step — don't guess. If you genuinely need an action type the catalog doesn't have, that's a Duhem engine change, not something to invent locally; don't silently mint new `uses:` strings. Common shape (terse): ```yaml - id: AC-1.1 description: steps: - id: create description: Create the workspace through the public API uses: api/call with: { method: POST, url: $inputs.api_base/workspaces, timeout: 3s } # no outputs: block — status / body / body.id resolve directly assertions: - $steps.create.outputs.status == 200 - type_check: { value: $steps.create.outputs.body.id, is: uuid } ``` Authoring rules: - Reference outputs by their fully-qualified path, `$steps..outputs.`; add `outputs:` only for a rename or a deep-extraction alias. - Use optional step `description:` for reader-facing intent. It labels reports ahead of `id` and ` #`; keep `id` terse and stable because references, diagnostics, and dashboard links use it. - Timeouts (`timeout:`) are explicit on steps that observe something asynchronous. - Steps default to `if: success`: an earlier execution `Error` / `Timeout` or failed implicit judgment gates the remaining default steps in that check. Binding a judging step's `satisfied` output opts out of both implicit judgment and gating, so a false observation can still participate in a later disjunction. Explicit `assertions:` are evaluated after the step sequence and cannot gate steps. Use `if: always` for teardown and `if: failure` for diagnostics that should run only after failure. Both still run after a step-level engine error before the original error aborts the run. A retried check runs its cleanup once per attempt, so cleanup steps must be idempotent. Do not invent expressions under `if:`; its closed vocabulary is `success | always | failure`. - Use role-based locators (`{ role: "button", name: "…" }`) for `ui/*` rather than CSS or XPath — UI churn invalidates the latter while role-based selectors track the user-visible affordance. - A check with no verdict — neither `assertions:` nor a judging step — is a script, not a check. Reject it. - The `capture/` output-name prefix is reserved for runner-emitted failure evidence (a failing `ui/*` check auto-records a screenshot, DOM, and network HAR). Captures are evidence for humans/agents, never judge input; an authored output under `capture/` is rejected at validate time. #### Share locators instead of copying them When two leaves use the same locator, declare it once in an ordinary manifest include and reference it as `$pages..`: ```yaml # .duhem/pages.yml (included by .duhem/duhem.yml) pages: login: username: { role: textbox, name: Username } submit: { role: button, name: Sign In } chat: history_item: { xpath: '(//article)[{}]' } # a leaf step - uses: ui/type with: { locator: $pages.login.username, text: $inputs.user } - uses: ui/click with: { locator: $pages.chat.history_item($inputs.index) } ``` The catalog is flat below each group; do not add a `locators:` layer. A leaf-local entry wins over a shared entry of the same name. Catalog entries may use `$inputs.*` for dynamic scope text, but every leaf that receives that catalog must declare each referenced input. `duhem validate` catches dangling `$pages.*` names offline and suggests nearby entries; `duhem resolve --provenance` lists the effective catalog and the source file of each winning entry. For a selector template, call the two-segment reference as `$pages..(args…)`: arguments are ordinary expressions and fill `{}` placeholders positionally across the entry's string values. The argument count must exactly match the entry's placeholder count or `duhem validate` rejects the call; use `{{` / `}}` for literal braces. A bare reference to an entry containing `{}` is an arity error. #### Share step flows instead of copying sequences Put a repeated action sequence in a `flows:` catalog on the root manifest, an included fragment, or a leaf. Parameters are bound at each `call:` and reuse the input type catalog: ```yaml # flows.yml flows: sign_in: description: Sign in with supplied credentials params: password: { type: string, secret: true } user: { type: string } steps: - uses: ui/type with: { locator: $pages.login.username, text: $params.user } - uses: ui/click with: { locator: $pages.login.submit } - id: landed uses: ui/assert-element with: { locator: $pages.login.welcome, expected: visible } outputs: signed_in: $steps.landed.outputs.satisfied # leaf check - id: login description: Sign in as the fixture user call: sign_in with: { user: $inputs.user, password: $inputs.password } ``` A flow body is hygienic: step `with:` values may reference only `$params.*` and `$pages.*`, never the caller's `$inputs.*` or `$steps.*`. Pass every dependency explicitly. The one intentional `$steps.*` use is in the flow's own `outputs:` map, which defines the entire caller-visible interface. Callers read only `$steps..outputs.`; inner step ids are private. `secret: true` parameters receive the same evidence masking as secret inputs. Use `duhem resolve --provenance` to inspect the expanded action sequence and its flow origins. ### 4. The holistic-environment tax — no mocks of the web A Duhem check exercises real behavior end-to-end. **No mocking the web.** - There is no `api/mock` or `db/stub` action — they don't exist by design. - Don't write a check that runs against an in-memory test double of any subsystem the artifact depends on. - If a check needs data preconditions (a seeded user, a pre-existing record), use `db/seed` (or an equivalent setup action) in the verification's `setup:` block, against the **real** database. Use `setup:` for once-per-verification preconditions; don't duplicate them inside every check. If verifying the criterion would require mocking the web, that's a signal either the criterion's assumptions are unstated (reformulate it against the real web) or the check is the wrong shape. Don't paper over it with a mock — stop and reconsider the criterion. #### Bring the system up first with `provision:` Most real checks need the system under test running before any check observes it. Declare `provision:` on the VD or the suite manifest (one shared environment for the whole suite) instead of assuming it's already up: ```yaml provision: up: ./scripts/up.sh # required — stands up the real thing, no mocks down: ./scripts/down.sh # optional teardown, runs regardless of verdict ready: http: { url: http://127.0.0.1:8080/health, timeout: 120s } ``` Duhem forks `up:` once before `setup:`, polls the optional `ready:` probe, runs every check against the now-real environment, then forks `down:` after the criteria loop completes. If the environment never comes up, the run is `inconclusive` — never a false `pass`. Teardown failures are recorded as evidence and don't flip the verdict. ### 5. Mechanical judgment — no LLM in the verdict Every assertion must be a deterministic predicate the judge can evaluate. Allowed forms: - Boolean expression: `$steps.X.outputs.Y == 200` - Type check: `type_check: { value: …, is: uuid|string|integer|float|boolean|object|array|null }` - Pattern match: `$runtime.matches(value, "regex")` - Membership: `$runtime.contains(haystack, needle)` — literal substring on a string, element membership on an array - Existence: `exists: $steps.X.outputs.Y` - Cross-step consistency: `equal: [$steps.A.outputs.X, $steps.B.outputs.X]` Things that look like assertions but aren't: "the response makes sense", "an LLM grades the output", "the screenshot looks right". None are mechanical; none are allowed. A type error in an assertion (e.g. `contains` against a number) is a `fail` that names the mismatch, not a silent pass. If a criterion seems to require non-mechanical judgment, split it until each piece is mechanical, or note explicitly that it can't be Duhem-verified today — don't paper over it. ### 6. Self-validate Before saving, walk through — then let `duhem validate` close the loop: - [ ] Each criterion appears verbatim under `criteria:` with an `id:` (`AC-1`, `AC-2`, …). - [ ] Each criterion has at least one check. - [ ] Each check has a non-empty `steps:` and a verdict (an `assertions:` block **or** a judging step). - [ ] Every referenced output is one the action declares (or is bound via `outputs:` for a rename/extraction). - [ ] Every assertion is one of the allowed mechanical forms. - [ ] No mock action types; no LLM-grading anywhere. - [ ] Action types come from the catalog (`duhem actions`). ### 7. Save and register The file can be named anything (it self-identifies via a top-level `verification:` / `criteria:`). Conventional names: `.verification.yml`, `.yml`, `verification.yml`. If the suite uses a `.duhem/duhem.yml` manifest, add the file to `verifications:`; a standalone file run via `duhem run ` needs no manifest entry. Running a leaf by path still resolves shared context: `duhem run ` walks that file's own ancestors (capped at the enclosing `.git`) for a root manifest, the same search the no-path form uses, and merges its `pages:`/`flows:` catalog into the leaf so `$pages.*` and `call:` references still resolve. Sibling `verifications:` entries the manifest lists are not loaded or run — the run stays scoped to just the requested leaf. A leaf with an unresolved `$pages.*` or `call:` name and no discoverable manifest fails at validate/run time naming exactly what it needs. ### Secret inputs and acquired credentials Never commit a credential as an input `default:` or pass it in a long-lived command line. Declare its process-environment source and mark it secret: ```yaml inputs: db_dsn: type: string env: DATABASE_URL secret: true ``` Resolution is `--inputs` → selected profile → declared `env:` → `default:`. Registered secret values, plus their base64, percent-encoded, and JSON-string-escaped forms, are exact-substring masked from recorded text and terminal output. Screenshots/video are not masked, and other transformations do not match, so do not render credentials into the product UI. Prefer high-entropy fixture values; short/common values can over-mask the evidence bundle and produce a run-time warning. When multiple Verification Definitions share a credential, declare it once in the suite manifest and let each consumer inherit the name: ```yaml # .duhem/duhem.yml inputs: db_dsn: type: string env: DATABASE_URL secret: true verifications: - path: database/duhem.yml # .duhem/database/duhem.yml inputs: db_dsn: { inherit: true, secret: true } ``` The manifest declaration owns type checking, `env:`, and `default:`; the leaf may add `secret: true`, and selected `profiles:` entries continue to supply values. Resolution is `--inputs` → selected profile → manifest `env:` → manifest `default:`. An inherited leaf declaration cannot carry `type:` or `default:`. When a login or setup action returns a credential, declare its scalar output path on the producing step: ```yaml - id: login uses: api/call secret_outputs: [body.data] with: method: POST url: $inputs.login_url body: { username: $inputs.username, password: $inputs.password } - id: read uses: api/call with: method: GET url: $inputs.projects_url headers: { Authorization: $steps.login.outputs.body.data } ``` `secret_outputs:` paths start at an output declared by the action contract and must resolve to one scalar. `body.data` and `body.items[0].key` are valid scalar leaves; `body` when it is an object and `body.items` when it is an array are errors. Do not name a whole response subtree: exact serialization masking almost never matches later evidence, while masking every leaf can swallow unrelated values. Registration happens before the producing step records its response, so its own observations and later references become `[redacted:login.body.data]`. An action may also mark a credential output secret in its contract, requiring no authored `secret_outputs:` entry. ### Reuse an acquired browser session Authenticate once in `setup:`, capture the resulting Playwright storage state, and seed each authenticated check explicitly: ```yaml setup: - uses: ui/navigate with: { url: "$inputs.base_url/login" } - uses: ui/type with: { locator: { label: Password }, text: $inputs.password } - uses: ui/click with: { locator: { role: button, name: Sign in } } - id: session uses: ui/capture-session criteria: - id: AC-1 description: An authenticated user sees the workspace list. checks: - id: AC-1.1 session: $setup.session.outputs.state steps: - uses: ui/navigate with: { url: "$inputs.base_url/workspaces" } - uses: ui/assert-element with: locator: { role: heading, name: Workspaces } expected: visible ``` `session:` must be one whole-string `$` reference. Do not hand-author cookies or local storage: acquire the state by exercising the real login surface. Each check still receives a fresh browser context seeded from that baseline, so mutations do not cross between checks. Omit `session:` for the signed-out path. `ui/capture-session` marks its structured `state` output secret by contract; do not add an authored `secret_outputs:` entry for it. ## Worked example template ```yaml verification: spec_ref: inputs: api_base: type: string default: https://staging.example.com setup: # once-per-verification preconditions — real environment, no mocks - uses: db/seed with: table: users rows: - { id: $runtime.uuid(), email: "test@example.com" } criteria: - id: AC-1 description: | checks: - id: AC-1.1 description: steps: - id: uses: with: { … } # add outputs: only to rename or deep-extract a field assertions: - .outputs.> ``` ## Anti-patterns (don't) - **Criteria with implementation details.** "Click the button at `#create-workspace`" is a check pretending to be a criterion. Rewrite as "A user can create a workspace from the dashboard." - **A check with no verdict.** Neither an `assertions:` block nor a judging step — it's a recording, not a check. - **Redundant identity bindings.** `outputs: { foo: foo }` does nothing — reference `$steps..outputs.foo` directly. `outputs:` is for a *rename* or a *deep extraction*, not for re-declaring a native output. - **Mocks anywhere.** No `api/mock`, no `db/stub`, no time-freeze short-circuits. The web Duhem verifies is the real one. - **LLM-grading assertions.** Non-mechanical judgment is out. - **One mega-criterion.** "User can create, list, edit, archive, and delete workspaces" is five criteria. - **Re-authoring frozen checks every run.** Checks are authored once, reviewed, then frozen. Regenerating them from scratch reintroduces the drift Duhem exists to prevent. - **Verifications that smuggle in unit tests.** A check that exercises a single function in isolation isn't holistic — expand its scope to a real web slice or remove it. ## References - `duhem actions` / `duhem describe ` — the version-exact action catalog and per-action contract. Author against these, not memory. - `duhem validate ` — field-checks your VD and names valid options on a miss. - `duhem resolve --provenance` — prints the merged, resolved document without running it. Use this when an include, profile, inherited input, or default produced a surprising value: ```bash duhem resolve .duhem --profile staging --provenance duhem resolve .duhem --profile staging --format json > resolved.json ``` Secrets remain `••••••`, and validation failures appear in the output. The command has no browser, provisioning, evidence-write, or network side effects. - `duhem run ` — runs the suite (exit 0 == pass). Add `--reporter json` for a per-criterion verdict breakdown; the default reporter prints only the overall pass/fail. - `duhem mcp` — exposes describe/actions/validate to your coding agent over MCP for AI-assisted authoring. - The Duhem documentation for the Holistic Verification Principle, the criteria-vs-checks separation, and the full Verification Definition schema.