colinhacks real-subsystem-harness — spin the real thing, assert the real function
Browser QA can't reach the broker socket, a real pty, the resume/wake path, SQLite migrations, or the
scheduler. For those, write a small nub harness that spins the real resource and asserts the
real function — a mock proves only that your mock matches your belief.
Worked examples in this repo: scripts/verify-login-pty.mjs (a real node-pty behind the login transport,
with a negative control), scripts/win-claude-resolve-probe.mjs (the real binary resolver, with a
differential control that proves the tool under test is actually installed), and
scripts/verify-orphan-reaper.mjs.
import { execFileSync } from "node:child_process"
import { theFixedFunction } from "../packages/server/src/<module>.ts"
// 1. create the real precondition (a real pty, a real socket, a real sqlite db, a real server…)
// 2. call the real function
// 3. PASS/FAIL each assertion to stdout; process.exit(1) on any failure
// 4. tear the real resource down in finally
The rules that make a harness worth running
Include a NEGATIVE CONTROL. Not just the happy path — an identity-mismatch case that must still be rejected. That is what proves a widened code path didn't weaken a safety check. And it is what proves the harness itself can fail: a suite that has never gone red is not evidence. When an experiment confirms what you expected, get suspicious rather than relieved — check that you varied exactly one thing, that the command actually ran, and that you are not testing a stale build.
Replicate production faithfully. verify-legacy-wake.mjs caught a trailing-quote boundary bug in a
matcher on its first run because it used the production argv form instead of a hand-quoted string. A
hand-driven proxy — invoking a CLI yourself with the flags the server would have passed — proves the
parts, not the whole. If a feature spawns, injects, or renders something, drive the REAL spawned thing.
Test the whole, not the parts. A passing unit test, a mock, or a typecheck proves the pieces. The seam between the pieces is exactly where the bug lives. "I verified the components" is how a broken feature ships.
Poll with an EARLY EXIT — and always one for the failure case. A loop that only breaks on success burns its entire budget every time the answer is "no". A 150×2s wait cost five minutes of dead clock on a control run that was supposed to fail. Break on the negative signal too (the worker rested, the process exited, the thread went idle), not only the positive one, and log what you are waiting for.
Never read an exit code through a pipe. cmd | head then echo $? gives head's status. Redirect,
capture on its own line, then inspect: cmd > /tmp/out.log 2>&1; echo "EXIT=$?". Two related traps:
| head -N can SIGPIPE-kill the producer mid-run, and a background task's completion notification
reports the PIPELINE's exit — so a failing run can arrive labelled "exit code 0".
Clean up by exact PID. Anything you spawn that outlives the harness — a detached daemon above all —
is yours to kill, by the exact pid you started, never a broad pkill -f. Other agents share this machine.
Verify the survivors afterward; an empty variable in a grep pattern matches EVERYTHING.
Say what you could not verify. If genuine end-to-end testing is infeasible, that does not lower the bar — it raises it: attack your own assumptions, enumerate how the change could fail in the real runtime, and trace the whole path yourself. Then state plainly what you exercised and what you did not. "It should work" is not "it works".
Composes with
frizz-stack— when the real resource is a running Frizz (a real server, real tenants, a real dispatched worker), boot it there rather than hand-rolling one.frizz-artifact-e2e— when the behavior could differ between dev source and the promoted artifact.headless-browser— for the half of the system a harness cannot see.