# Watch Run

> Emit-on-event health watcher for a long paired/ab benchmark run, meant to be fed to the Monitor tool (each stdout line becomes one notification) so an unattended 75-min run cannot fail silently. Coverage-first by design — it speaks up on liveness (ALIVE), on new error signatures in the proxy/tee logs (ERROR), on a stall or crash (STALL/DEAD — silence is not success), and on completion (COMPLETE) — not just the happy path. Stall is tracked on the SHADOW (A0) turn-log, which is passthrough with keep-alive OFF so every line is a real turn; the PRIMARY (A5) log also grows from keep-alive pings during idle gaps and would mask a stalled conversation, so it must not drive stall detection. Use it to babysit `.skills/paired` or `.skills/ab` runs whose wall clock exceeds the Monitor's 60-min non-persistent cap (run it persistent).

- Skill: `zapgun-ai/watch-run` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add zapgun-ai/watch-run`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zapgun-ai/watch-run/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: zapgun-ai (https://skillmd.com/u/zapgun-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zapgun-ai/watch-run

---


# clawback run health watcher

`.skills/watch-run/scripts/watch_run.sh <out-dir> [listen-port primary-port shadow-port]`

Defaults: ports `8788` (tee/listen) `8790` (primary) `8791` (shadow) — the
`.skills/paired` defaults. Pass the run's `--out` dir as the first arg.

## How to use (with Monitor)

Launch the run in the background first (e.g. `.skills/scripts/run_paired.sh … --out
runs/paired-haiku-L1`), then attach this under the **Monitor** tool with
`persistent: true` (a 75-min run exceeds Monitor's 60-min non-persistent
timeout). The watcher exits on its own at COMPLETE/DEAD/WATCH-END, which ends
the Monitor cleanly; TaskStop it early if you abort the run.

```
Monitor(persistent=true,
  command=".skills/watch-run/scripts/watch_run.sh runs/paired-haiku-L1 8788 8790 8791")
```

## Events it emits (one line = one notification)

- `ALIVE a0=.. a5=..` — first turns observed; the run came up.
- `ERROR(+n) … :: <last matching line>` — n new error-signature lines in
  `proxy.primary.log` / `proxy.shadow.log` / `tee.log` (overload, 429,
  EADDRINUSE/ECONN*, `→ 4xx/5xx`, Traceback, force-exit, `[error]`, …).
- `STALL …` — no new SHADOW turn for 12 min but a proxy is still listening
  (warned once; the conversation is wedged, processes alive).
- `DEAD …` — no new SHADOW turn for 12 min AND no proxy listening → the run
  crashed; exits 1.
- `COMPLETE a0=.. a5=.. (+Nm)` — `report.md` present (analyzer finished);
  exits 0.
- `WATCH-END` — ~95-min wall-clock backstop tripped; exits 2.

## Why shadow-tracked stall (the subtle bit)

On a paired run the PRIMARY arm (A5 stack) fires keep-alive **pings** during
idle gaps, each appended to `instance.A5.ndjson`. So A5's line count rises even
when no real turn happens — tracking it for staleness would hide a stalled
conversation behind ping traffic. The SHADOW (A0) is `--passthrough` with
keep-alive off, so its line count == real-turn count: the honest liveness
signal. `ALIVE` still fires on either arm (proxies-up signal); only stall/crash
detection keys on A0.

## Tuning

Both are **env vars** (soft default + knob), so tune them per profile without
editing the script — pass them on the Monitor command line:

```
STALL_SEC=2400 .skills/watch-run/scripts/watch_run.sh runs/paired-haiku-L2 8788 8790 8791
```

- `STALL_SEC` (env, default 720s ≈ 12 min) — raise for L2+ where long gaps are
  normal: `STALL_SEC=2400` for L2 (up-to-30-min gaps), `~6000` for L3 (90-min).
  Set it above the profile's max gap + response slack. Too-low only emits a
  STALL warning; it never causes a false DEAD (that requires no proxy
  listening), so erring high is safe.
- `HARD_MAX_SEC` (env, default 5700s ≈ 95 min) — raise for runs capped longer
  than 75 min.

## Self-check (error detection is the fragile part)

The error-signature pipeline misfired twice in production — a startup crash
(`grep -c`'s print-0-and-exit-1 double-counted into `0\n0: syntax error`) and a
false ERROR on a healthy `.429Z` keep-alive ping (a bare `429` matched the
timestamp's milliseconds). Both are fixed and the detection now lives in one
tested function reachable via a hidden entry point:

    .skills/watch-run/scripts/watch_run.sh --errcount <logfile>   # prints the error count for one file

`test/watch_run.test.js` pins the must-NOT-cry-wolf cases (the `.429Z` ping, the
`→ 404 [no-route]` probe, a 4-digit `→ 4096` count) and the must-NOT-go-blind
cases (`→ 529`, `→ 5xx`, `EADDRINUSE`, `Traceback`, `[error]`). After touching
`ERRRE`/`IGNORE`, run `npm test -- watch_run`.