clawback run health watcher
.skills/watch-run/scripts/watch_run.sh <out-dir> [listen-port primary-port shadow-port]
Defaults: ports 8788 (tee/listen) 8790 (primary) 8791 (shadow) — the
.skills/paired defaults. Pass the run's --out dir as the first arg.
How to use (with Monitor)
Launch the run in the background first (e.g. .skills/scripts/run_paired.sh … --out runs/paired-haiku-L1), then attach this under the Monitor tool with
persistent: true (a 75-min run exceeds Monitor's 60-min non-persistent
timeout). The watcher exits on its own at COMPLETE/DEAD/WATCH-END, which ends
the Monitor cleanly; TaskStop it early if you abort the run.
Monitor(persistent=true,
command=".skills/watch-run/scripts/watch_run.sh runs/paired-haiku-L1 8788 8790 8791")
Events it emits (one line = one notification)
ALIVE a0=.. a5=..— first turns observed; the run came up.ERROR(+n) … :: <last matching line>— n new error-signature lines inproxy.primary.log/proxy.shadow.log/tee.log(overload, 429, EADDRINUSE/ECONN*,→ 4xx/5xx, Traceback, force-exit,[error], …).STALL …— no new SHADOW turn for 12 min but a proxy is still listening (warned once; the conversation is wedged, processes alive).DEAD …— no new SHADOW turn for 12 min AND no proxy listening → the run crashed; exits 1.COMPLETE a0=.. a5=.. (+Nm)—report.mdpresent (analyzer finished); exits 0.WATCH-END— ~95-min wall-clock backstop tripped; exits 2.
Why shadow-tracked stall (the subtle bit)
On a paired run the PRIMARY arm (A5 stack) fires keep-alive pings during
idle gaps, each appended to instance.A5.ndjson. So A5's line count rises even
when no real turn happens — tracking it for staleness would hide a stalled
conversation behind ping traffic. The SHADOW (A0) is --passthrough with
keep-alive off, so its line count == real-turn count: the honest liveness
signal. ALIVE still fires on either arm (proxies-up signal); only stall/crash
detection keys on A0.
Tuning
Both are env vars (soft default + knob), so tune them per profile without editing the script — pass them on the Monitor command line:
STALL_SEC=2400 .skills/watch-run/scripts/watch_run.sh runs/paired-haiku-L2 8788 8790 8791
STALL_SEC(env, default 720s ≈ 12 min) — raise for L2+ where long gaps are normal:STALL_SEC=2400for L2 (up-to-30-min gaps),~6000for L3 (90-min). Set it above the profile's max gap + response slack. Too-low only emits a STALL warning; it never causes a false DEAD (that requires no proxy listening), so erring high is safe.HARD_MAX_SEC(env, default 5700s ≈ 95 min) — raise for runs capped longer than 75 min.
Self-check (error detection is the fragile part)
The error-signature pipeline misfired twice in production — a startup crash
(grep -c's print-0-and-exit-1 double-counted into 0\n0: syntax error) and a
false ERROR on a healthy .429Z keep-alive ping (a bare 429 matched the
timestamp's milliseconds). Both are fixed and the detection now lives in one
tested function reachable via a hidden entry point:
.skills/watch-run/scripts/watch_run.sh --errcount <logfile> # prints the error count for one file
test/watch_run.test.js pins the must-NOT-cry-wolf cases (the .429Z ping, the
→ 404 [no-route] probe, a 4-digit → 4096 count) and the must-NOT-go-blind
cases (→ 529, → 5xx, EADDRINUSE, Traceback, [error]). After touching
ERRRE/IGNORE, run npm test -- watch_run.