gabrielmoreira Verification
Task completion is not goal achievement. Verify that the phase achieved its goal, not that prior agents said it did. If you cannot independently reproduce a claimed success, return FAIL.
Reference Files
references/live-production-testing.md— live/production verification strategy
The Gate Function
COMPLETION → TRUTH → PROOF
- Completion: Did the agent finish its task? (contract says PASS)
- Truth: Is the work actually correct? (independent verification)
- Proof: Can you prove it with evidence? (exit codes, test output, screenshots)
A PASS without proof is a claim. When a result surprises you in either direction — a pass that came too easily, or a failure you cannot explain — suspect the observation method before the system: confirm the check runs the right command against the current build and exercises the real code path before theorizing about the code.
Self-Critique Gate (BEFORE Verification Commands)
Before running any test, audit your own work:
Code Quality:
- No TODO/FIXME/stubs in changed files
- No commented-out code
- No debug logging left in
- Error handling covers the failure modes the change introduces
- Types are correct (not
as anyoras unknown)
Implementation Completeness:
- Every acceptance criterion from the plan has corresponding code
- Every named scenario has a test
- Every error path in the plan has handling
- No silent failures (empty catches, discarded errors)
- Build succeeds, type-check passes
Self-Critique Verdict: If any check fails, fix it BEFORE running verification. Don't run tests you know will fail on code quality issues — a known-failing run produces noise evidence that then has to be explained away.
Validation Levels
| Level | What | Exit Code | When |
|---|---|---|---|
| Deterministic | Automated test | 0/1 | Always — the default |
| Probabilistic | Test with known flake rate | 0/1 (retry policy) | When deterministic impossible (timing, network) |
| Manual | Human verifies with checklist | n/a | When automation not worth the cost |
| Live | Production-like environment | 0/1 | When plan requires live proof |
Every verification must state its validation level — unlabeled evidence lets weak proof masquerade as strong. If manual, state the checklist. If deterministic, state the command + exit code. If live, state the harness command.
Production-Like Live Proof
When the plan includes ### Live Verification Strategy or a harness manifest:
- Run
python3 "${CLAUDE_PLUGIN_ROOT}/tools/live_harness_runner.py" --manifest <path> --mode proof - If stress required: also run
--mode stress - Do NOT silently substitute replay fixtures or unit tests for required live proof
Flaky test handling: re-run once — one retry separates environment blips from real flake; more retries launder genuine failures. Pass on re-run → mark PASS with flaky: true. Fail both → FAIL. Never convert flaky pass into unconditional confidence.
Evidence Array Protocol (MANDATORY for PASS)
EVIDENCE:
scenarios:
- "[name] | Given [state] | When [action] | [command] → exit [code] | expected=[expected] | actual=[actual]"
regressions:
- "[test] → exit [code]: [result]"
edge_cases:
- "[case]: [command] → exit [code]: [result]"
Every scenario needs non-empty Expected and Actual. Every scenario maps to exactly one EVIDENCE entry. SCENARIOS_PASSED must equal EVIDENCE.scenarios with exit 0 + Result=PASS. Run each check against the revision being verified; if the code changes after a check, re-run the affected scenarios before citing them. Record the tested identity with the evidence: the commit SHA, plus a patch or digest when the tree is dirty; evidence that does not identify the revision it was produced from is unverified, not PASS.
Goal-Backward Lens
Walk backward from the goal to verify it was achieved:
- What was the goal? (re-read the plan's exit criteria)
- What would prove it? (name the specific evidence)
- Do I have that evidence? (run the verification)
- Does the evidence actually prove the goal? (not "tests pass" but "the tests test the right thing")
Common Failures
| Failure | What happens | Fix |
|---|---|---|
| False green | Test passes without exercising the real code path | Test Honesty Gates (see integration-verifier) |
| Tautological check | Expected value recomputed the way the code computes it, e.g. expect(add(a, b)).toBe(a + b) — passes by construction |
Expected values come from an independent source of truth: a known-good literal, a worked example, the spec |
| Scope skip | "All tests pass" but untested scenarios exist | Goal-backward lens: name every scenario, verify each |
| Stale evidence | "Tests pass" but you didn't run them this session | Re-run. Evidence must be from THIS session. |
| Claim without proof | "It works" with no command/exit code | Evidence array is mandatory for PASS |
| Pointer-as-proof | Path cited as PASS without re-opening it | Re-open the artifact; if it does not show the claim, downgrade or re-run. Never cite it as PASS |
| Untested claim | A scenario or claim the return never addresses | Mark it untested with its reason; never assume it passed |
| Environment escape | Test fails with env signal (command not found, ECONNREFUSED) | Classify as ENVIRONMENT not code. Mark BLOCKED. |
Excuses and Tells
If you catch yourself in the left column before a PASS, do the right column first.
| Excuse or tell | Reality |
|---|---|
| "Should work now" / "looks good" / "seems fine" / "probably" | RUN the verification command. "Should" is not evidence. |
| "I'm confident it works" | Confidence is not evidence. Exit code 0 is evidence. |
| "Builder reported success" / "the agent said it passed" | Verify independently. Agents can be wrong or sycophantic. |
| "The tests cover this" | Name the specific test — or it isn't covered. |
| "No regressions detected" | List what was tested — or nothing was. |
| "Tests passed before" (another session) | Stale evidence. Re-run in THIS session. |
| "It's just a typo fix" / "it's trivial" | Small changes break things too. Run the tests. |
| "I'll verify after commit" / committing without fresh evidence | Verify BEFORE commit. A broken commit is harder to revert. |
| "The build succeeded so it works" | Build success ≠ correct behavior. Run behavioral tests. |
| "I don't need to run the full suite" | If your change touches shared code, you do. |
| Satisfaction before seeing exit code 0 | Evidence first, satisfaction after. |
| Weakening an assertion to make a test pass | That launders the failure. Fix the code, not the test. |