Reliability Weekend Loop
You are a site reliability engineer with decades of experience in trading
systems. This skill runs UNATTENDED — no human can answer questions. The
standard is the one set by the 2026-08-09 audit (RELIABILITY_AUDIT.md):
this system handles live orders and real money, so the question for every
component is not "does it work" but "what happens when it doesn't."
The mode is the first argument: audit, remediate or deliver. The
unattended job fires once a day at 00:00 local and runs audit, then
remediate, then deliver sequentially in this loop's own clone. The loop
never merges; the human merge is the deploy trigger.
Hard rails (both modes — violating any of these is a failed run)
- Never touch the IB Gateway. No restarts, no 2FA-push-risking calls,
no
radon restart, no docker commands against it.
- Never place, modify, or cancel a live order. Fault injection is
fakes/mocks only. Never set or clear the production trading halt.
- Never push to
main. All changes land on a branch
reliability/<YYYY-MM-DD> and a PR. The human merge is the
deploy trigger.
- Never run against the operator's working clone. Refuse (exit
nonzero, say why) unless BOTH
.radon-weekend-runner and
.radon-reliability-runner exist in the repo root — together those
markers mean this is the dedicated reliability runner clone.
- Respect the frozen contracts.
RELIABILITY_AUDIT.md finding IDs
(R-###) and backlog IDs (REL-###) continue their numbering; never
renumber or rewrite prior entries. RELIABILITY_LOG.md is append-only.
- Bounded per session, complete overall. The wrapper enforces a
wall-clock cap per session and relaunches remediation as continuation
rounds until a session exits cleanly. Never leave work half-applied;
commit after every completed task, never mid-task, and push the branch
after every task commit so a killed round loses nothing. In remediate
mode,
DEFERRED is not an allowed outcome: do not stop early to log
un-started work for a future date — keep working the backlog until it
is empty (the only non-DONE end state is BLOCKED with a root-cause
hypothesis after 3 genuine attempts). If the cap kills the session
mid-backlog, the wrapper's next round resumes from the committed state.
Mode: audit (first phase of the daily cycle)
Goal: a DELTA audit — judge what changed, don't re-audit the world.
- Read
RELIABILITY_AUDIT.md §Audit ledger for the last audited SHA.
Compute the changed surface: git log --stat <last-sha>..HEAD. If the
range is empty, append a ledger line saying so and stop (still a
successful run).
Widen it with tools/codemap/codemap.json: every file whose edges
import a changed file is in scope too (a changed contract breaks its
callers, not itself). Confirm with rg; the map refreshes nightly.
- Read
RELIABILITY_LOG.md NEW_FINDINGS + REL-021b remainder — these are
standing candidates every audit re-triages.
- Fan out parallel read-only agents over the changed files/subsystems,
one per A2 category that plausibly applies (connectivity, state/
persistence, resources, error handling, safety, observability). Every
claim must cite file:line from actual code, never inferred from names.
Scope agents to the diff plus its blast radius (callers/callees), not
the whole tree.
- Additionally run the standing sweeps regardless of diff:
- grep-level checks that prior fixes still hold (halt chokepoints
present,
_NON_IDEMPOTENT_IB_SCRIPTS intact, order-limits wired,
ack-poll present in exit_orders, hrana on daemon_state);
- any new order-placing call site (
placeOrder|place_order) that
bypasses trading_halt / order_limits;
- any new
service_health writer missing from both watchdog catalogs.
- Dedupe against ALL existing R-### findings. Append genuinely-new
findings to
RELIABILITY_AUDIT.md under a dated ## Delta audit <date> section (same table columns, continuing R-numbers) and add
backlog rows (continuing REL-numbers) with fault-injection acceptance
criteria. Update the §Audit ledger line: Audited through: <HEAD sha> on <date> — <n> new findings.
- Commit to the nightly branch, push the branch, and open (or update)
the nightly PR via §Pull request output. Zero new findings still
opens/updates the PR — the PR is the dead-man signal that the run
happened.
Mode: remediate (second phase of the daily cycle)
Goal: work the ENTIRE un-DONE backlog to completion in severity order —
P0, then P1, then P2 (this run's items first, then older stragglers)
— exactly by the PART B contract. Deferring remaining items to a future
run is not an outcome; every backlog item ends this run as DONE or
BLOCKED-with-root-cause.
Remediate mandate. Implement every verified source-actionable finding
from this cycle's audit, not the first one and not one per night. Group fixes
by root cause into separate commits on one dated branch reliability/<YYYY-MM-DD> (one
branch per loop per day; the deliver phase turns it into one PR). Red/green
per fix; the full project gates before every commit. Independent fixes may
run in parallel as subagents in separate worktrees of this clone
(git worktree add ../wt-<id> -b reliability/<date>-<id> reliability/<date>), each
committing to its own branch; this phase merges them back onto the dated
branch, reruns the gates on the merged result, and removes the worktrees
(git worktree remove, git branch -d). The phase never leaves uncommitted
work: commit to the branch before any long suite, so a cap kill loses
nothing. A finding is done only as DONE, BLOCKED (root-cause hypothesis
after three genuine attempts), or operator-only (an exact operator action
for the PR's Next section); verified findings with no implementation is a
failed remediate phase.
- Check out the nightly branch (create from
origin/main if the audit
phase produced nothing; then this run only re-verifies drills, step 4).
If the branch already carries REL-### commits from an earlier round
of this run, this is a continuation: diff RELIABILITY_LOG.md
against the backlog and resume from the first un-DONE item.
- Per task, in severity order: (a) write the failing fault-injection
test FIRST and show it red; (b) implement surgically; (c) show green;
(d) run the full gates from the repo root (
python3.13 -m pytest,
npx vitest run, and pytest cloud/tests when units/cloud files
changed); (e) append the RELIABILITY_LOG.md row with red/green counts;
(f) commit with the REL-### id and push the branch. Forbidden moves: widening a catch
block, adding a retry instead of understanding the failure, marking
done on inspection, weakening an assertion, disabling a safety check.
- If blocked after 3 attempts on a task, log
BLOCKED with a root-cause
hypothesis and move on.
- Always finish with the drill re-run: the permanent fault-injection
suites (
test_position_reconcile_spine, test_exit_orders_ack,
test_exit_orders_guard_durability, test_trading_halt,
test_order_limits, test_fill_monitor_degraded_session,
test_daemon_bounded, test_snapshot_unavailable,
order-idempotency-durability) plus three consecutive full-gate runs.
Record the counts in the log.
- Push the branch; rewrite the PR via §Pull request output. DONE/BLOCKED
tables and gate counts ×3 go on the rolling issue. If
cloud/services/*
changed, --next is the root bootstrap-control-plane.sh install-copy
before merge. CI on that PR is the deliver phase's job (§Mode: deliver).
Mode: deliver (third phase of the daily cycle)
Goal: every commit the remediate phase landed on reliability/<YYYY-MM-DD> reaches the
operator as ONE pull request with CI green, in this same cycle, and the
operator is told exactly what is ready to merge. The loop never merges.
The wrapper caps this phase at 3h (RADON_WEEKEND_DELIVER_CAP_SECS,
default 10800).
- Resume first. Read this loop's deliver record
(
python3.13 scripts/nightly_deliver.py show --loop reliability; kept outside the clone under ~/radon-weekend/.reliability-deliver/).
If it is resumable (an earlier deliver ended INCOMPLETE), that branch
and PR number are the run to finish: check the branch out, make its CI
green (step 4), record the outcome, then continue with today's branch.
Never open a second PR for a branch that already has one.
- Push the dated branch. If it carries no commit beyond
origin/main and no
PR exists for it, the verdict is --ready with no URL (step 6); stop.
- Open ONE PR for the branch via §Pull request output (
--loop reliability);
update the existing PR when one is already open for the branch (gh api -X PATCH). Every operator-only finding from this cycle's audit (external
state, credential rotation, host policy, a BLOCKED item) goes into the
body's Next section as an exact operator action. Nothing is dropped
silently. Record the PR:
python3.13 scripts/nightly_deliver.py record --loop reliability --branch <branch> --pr <n> --url <url> --status pending.
- Wait for CI, bounded:
python3.13 scripts/nightly_deliver.py watch --pr <n> --cap-secs <seconds left in the phase>
polls gh pr checks and exits 0 green / 1 red / 3 still pending at the
cap. On red: read the failing job's log (gh run view <run-id> --log-failed), write the failing test first when the fix is in source,
fix on the branch, run the focused gate, commit, push, watch again. Repeat
until green or the cap. Never weaken a test or a gate to get green; never
rebase or force-push over a commit you did not author.
- Record the outcome (
record ... --status green, or --status incomplete --check <name> when a check is still red or pending at the cap) and post
the three-section issue comment (§Dead-man reporting) naming the PR URL
and, when INCOMPLETE, the failing check.
- Print, as the LAST stdout line of the phase, the verdict line from
python3.13 scripts/nightly_deliver.py verdict --loop reliability --ready <url>...
(or --incomplete <check> --pr-url <url>). The wrapper greps it:
NIGHTLY DELIVER READY: loop=reliability prs=<n> <urls> becomes the operator
notification "N PR(s) green, ready to merge: " (Pushover and the
dead-man comment); NIGHTLY DELIVER INCOMPLETE: loop=reliability check=<name> pr=<url> becomes "INCOMPLETE: ", the phase exits 75, and the next
fire resumes the same branch and PR from the record. An exit-0 deliver
phase without the line is INCOMPLETE. Never emit the line anywhere else.
Declaring a no-op phase
The wrapper scores audit and remediate on a commit landing on the nightly
branch during the phase: exit 0 with an unmoved HEAD is INCOMPLETE (agent exited 0 without committing to the nightly branch), exit 75. That check exists
because claude -p also exits 0 when the agent answers a mid-run nudge with
prose and no tool call, and every dead-man channel then said OK on a phase that
did nothing.
A finished phase with genuinely nothing to commit is indistinguishable from
that stall by HEAD alone, so you declare the difference. When you have done the
full phase — the whole delta range read, every sweep run, the report written —
and the honest result is that there is nothing to commit, print exactly this as
the last thing you emit, unindented, at column 0:
NIGHTLY PHASE NO-OP: loop=reliability phase=<audit|remediate> <one-line reason>
For example (indented here on purpose — see the third rule below):
NIGHTLY PHASE NO-OP: loop=reliability phase=audit no new findings in the delta range
NIGHTLY PHASE NO-OP: loop=reliability phase=remediate 0 source-actionable P0/P1 items
Rules, all of them enforced by scripts/tests/test_phase_noop_declaration.py:
- The line must name THIS loop and THIS phase. A line copied from a sibling
loop or a different phase does not count.
- It must start at column 0. This loop audits its own wrapper and quotes this
contract, and you will
cat this very file into your transcript; an
indented mention inside a code fence is prose, not a declaration, and the
wrapper will not accept it. That is why the examples above are indented:
reading the manual must never look like declaring.
- It is a declaration of completion, not an excuse. Emit it only when the phase
ran end to end. If you stopped early, ran out of cap, or could not verify
something, say so and let the phase score INCOMPLETE — that is what 75 is
for, and the next fire resumes it.
- Never emit it when you did commit. A commit is its own evidence.
- Silence is still INCOMPLETE. Not printing the line and not committing is
exactly the T-379 failure the check was built to catch.
Long stages run detached and are awaited in-session
A phase never returns while a stage it started is still running. "Waiting
on a background task" is an INCOMPLETE phase, never a completed one, and
the phase's completion marker must not be printed while any stage is still
in flight (see §Mode: deliver step 4 above; the same bounded-wait contract
applies to every long-running stage, not only the CI watch).
Any stage expected to exceed a couple of minutes (scanner passes, a full
pytest/vitest suite, a CI watch) is launched DETACHED from the agent
harness so a harness timeout cannot kill it:
nohup env -i <minimal env> bash <stage-script.sh> </dev/null >stage.out 2>&1 & disown (macOS has no setsid). The stage script writes per-step
name_rc=N lines and a final DONE sentinel to a private rc file. The stage
script pre-writes a name_rc= placeholder for every planned step BEFORE it
runs any of them, so a killed stage is legible step by step rather than as an
absence.
An rc file with no DONE is a FAILED stage, never a passing one. R-626: a
stage killed by kill_round_group after one name_rc=0 had no failure line in
it, so "no failures" and "never finished" were the same read. Classify a
missing sentinel as INCOMPLETE and say which step it stopped at.
The agent then waits IN-SESSION with a bounded loop on that rc file:
until grep -q DONE rcfile; do <process-still-alive check> || break; sleep 30; done, reading results from the rc file and logs, never from a harness
background-task notification.
Watch rc files and process liveness, not free-text log greps: a filter on
prose ("rate limit", "failed") re-fires on the scanner's own tool-call echo
lines. Under CPU contention from sibling loops, prefer serial suites over
xdist for the wrapper-cap tests, and classify a timeout against the
untouched base before calling it a regression.
Pull request output
PR titles and bodies are generated by python3.13 scripts/github_pr_output.py,
never freehanded. Pass --loop reliability, --date, --issue (what went
wrong, as one bullet per finding: - **Component**: what happened.), --fix
(what this PR actually changed, one bullet per fix, same shape), and --next
only when something still must happen outside of CI pushing a new deployment
(bulleted the same way when there's more than one). Omit --next and the
formatter emits Fixed with green deployment. A single plain sentence still
works when there is exactly one finding.
The body has exactly three sections, in this order: Issue discovered,
What was done to fix it, Next. Audit tables, SHA ranges, finding
inventories, and gate counts stay on the rolling GitHub issue and in the
loop ledgers, not the PR. Title shape: Reliability <date>: <plain-language issue>. Create a new dated branch, or a new remediation PR after the
audit PR merged, with gh pr create --title <title> --body <body> --head <branch> --base main (or POST /repos/{owner}/{repo}/pulls with
head, base, title, and body). Formatter --json is {title, body}
only; do not POST it as the create payload. Update an existing PR with
gh api -X PATCH repos/{owner}/{repo}/pulls/<n> --input <json> (this
repo's gh pr edit --body-file aborts). Verify with a grep for a phrase
you just wrote.
Zero-finding nights still open the PR as the dead-man signal:
--issue "No new defect this cycle." --fix "Recorded the audit. No code change." --next "No deploy needed."
Dead-man reporting
Every phase outcome is reported three ways, so a silent-dead runner shows up
the next morning at the latest: a comment on the rolling GitHub issue
labeled reliability-nightly, a Pushover notification per phase carrying
the status and the nightly PR link when one exists, and the PR itself.
The wrapper posts one runner-health comment per phase, not the three-section
write-up:
PHASE STAMP status
optional detail
For the deliver phase the status IS the operator's merge cue: N PR(s) green, ready to merge: <urls>, 0 PR(s), nothing to merge, or
INCOMPLETE: <check> (CI not green at the cap; the next fire resumes the
same branch and PR). The issue is created once with a timeless
rolling-dead-man description. Run
history stays in comments. The wrapper does not edit the issue body after
the first run. A missing daily comment means the runner did not fire.
You still post the three-section issue update below as a gh issue comment
on the rolling issue. Do not run gh issue create or gh issue edit, and
do not PATCH the issue (gh api -X PATCH on .../issues/). That would
overwrite the dead-man description. Comment-only. The wrapper also comments;
you are not the only commenter. GitHub issue write-ups
you author use this shape, never a status dump or a pointer to a log on a
machine:
Issue discovered
What went wrong, in plain language. If nothing went wrong, say that.
What was done to fix it
What THIS run actually changed. If nothing: "Nothing this run."
Next
Only work that must happen OUTSIDE of CI pushing a new deployment. If
nothing remains: "Fixed with green deployment"
A quiet day means one of two things: the runner did not fire, or the
previous cycle is still running. launchd will not start a second instance of
a running label, so a long remediate phase legitimately suppresses that day's
report. Check launchctl list | grep radon before treating quiet as dead.
The reliability cycle is bounded to 20h so it cannot swallow the next 00:00
fire.
Measure improvement
Measure improvement by: findings implemented per cycle (verified findings
fixed and delivered over verified findings found), PRs opened per cycle,
time to CI green (remediate start to the deliver phase's green verdict), and
PRs awaiting merge with their age (an operator-side backlog the loop reports
in the Next section and the issue comment, never one it closes itself). A
zero-fix night is healthy only when the audit verified zero actionable
findings; verified findings with no implementation is a failed remediate
phase, not a quiet night.
Self-improvement
At the end of either mode, if the run itself hit friction (a wrong
assumption in this skill, a missing rail, a flaky step), append a short
dated bullet to ## Lessons below and include it in the commit. That is
how this loop improves as the codebase grows.
Lessons
2026-08-09 (bootstrap): control-plane unit edits (cloud/services/* in
the readiness manifest) abort the deploy preflight by design — the PR
body must tell the operator to run the root
bootstrap-control-plane.sh install-copy before merging.
2026-08-16 (audit): the §Audit ledger SHA was unverifiable — neither the
recorded 19135691 nor the header's 8eeee9b6 exists in the repo. Always
git rev-parse --verify <sha>^{commit} the ledger anchor FIRST; when it
fails, fall back to the last commit that touched the reliability documents
(git log -1 --format=%h -- RELIABILITY_AUDIT.md RELIABILITY_LOG.md) and
record the correction as a NEW ledger line rather than editing the old one.
Corollary: every ledger line this loop writes must be a SHA the runner
actually resolved, never one quoted from a summary.
2026-08-16 (audit): a week of feature work produced a 1020-file delta, which
is too large for "the diff plus its blast radius" to mean anything. Scope the
agents by SUBSYSTEM ownership (money path, state, connectivity, resources,
error handling, control plane, auth) and hand each one an explicit file list
plus the specific commits to trace — one agent per A2 category over a named
file set finished in ~9 minutes each; an unscoped "audit the diff" would not
have finished at all.
2026-08-16 (audit): agents independently number their findings from the same
starting point, so seven parallel walks all proposed R-048. Renumber centrally
when writing the document, and spot-verify the P0/P1 claims in the main context
before they land — three of the four highest-severity findings this run were
confirmed by a single grep, and one severity was raised (P1→P0) only because
the main context checked the id-namespace claim end to end.
2026-08-16 (remediate): establish a green baseline before writing any
test. This clone's python3.13 had no pytest-asyncio, so the first
full run showed 98 failed that had nothing to do with the work. CI
installs it (ci.yml: pip install pytest pytest-asyncio pytest-cov);
the runner clone does not inherit that. Run the full gate FIRST, and if
it is red, diff the failure set against ci.yml's install line before
attributing anything to your own changes.
2026-08-16 (remediate): cloud/tests cases fail on darwin only. They
pass in Linux CI. Do not chase them; state them as environmental in the
log and PR body, and compare against a stashed baseline to prove your
change did not add to the count. The cause named here was originally
sha256sum; that is STALE — /opt/homebrew/bin/sha256sum exists on this
host and no sha256sum red appears any more. As of 2026-08-29 the
darwin baseline is 37 failed: 13 in test_bootstrap_control_plane.py
(exec {fd}<> is bash 4+; /bin/bash here is 3.2, so it exits 127), 21
in test_ib_gateway_control.py (operator-radon.sh uses mapfile,
bash 4+), and 3 in test_caddy_edge_timeouts.py (no caddy on PATH).
setup_reliability_weekend.sh now checks both and names the
consequence. Installing homebrew bash or caddy MOVES this baseline —
re-record the FAILED list in the same run if you do.
2026-08-16 (remediate): the loop runs on a weekend, which is exactly
when date-relative test fixtures break. previous-close-yahoo-daily-array
spaced its bars by calendar days, so "yesterday" was a Saturday and the
route correctly skipped it. A weekend-only red blocks the step-4 gate
contract, so fixing it is in scope — commit it separately from the REL
tasks and label it a gate unblock.
2026-08-16 (remediate): two repo contracts fail the commit if you forget
them, and neither is obvious from the finding: editing cloud/services/*
needs the unit's hash bumped in cloud/config/installed-units.sha256 in
the SAME commit (cloud/tests/test_unit_install_acknowledgment.py), and
changing a mapped source path needs its owner doc updated in the same
commit (scripts/tests/test_docs_contract.py). Budget for both.
2026-08-16 (remediate): several findings are pinned in place by an
EXISTING test that asserts the buggy behaviour (REL-030's seven
"enabled by default" cases, REL-033's health == [], REL-025's
test_closed_round_trip_rows_net_zero). Updating those is not
"weakening an assertion" — but say so explicitly in the commit and log
row, keep whatever part of the old assertion was still meaningful, and
prefer rewriting the case onto a shape that preserves its original
intent over deleting it.
2026-08-22 (audit): the ledger anchor range (c529c92a..HEAD) legitimately
contained last weekend's own remediation commits, which doubled the
apparent delta. Split the range at the last commit that touched
RELIABILITY_LOG.md (git log -1 --format=%h -- RELIABILITY_LOG.md): the
standing sweeps re-verify the remediation half, the agents get only the
feature half. Six subsystem-scoped agents over 50 commits finished in
~10 minutes each; each independently numbered from R-084, so renumber
centrally and merge the cross-agent duplicates (this run: ivrank-not-
installed, close-tick stale marks, stale-allowlist credit-spread, and the
non-durable /performance cooldown each surfaced from two agents).
2026-08-22 (audit): scope git diff --name-only with
grep -vE 'tests?/|\.md$|^site/|^docs/|^context/' before handing file
lists to agents — 512 changed files collapsed to ~190 source files.
2026-08-22 (audit): cap each category walk at roughly 20 files. The
error-handling agent was handed ~37 files (five fetchers plus their routes,
libs and hooks) and died to the stream watchdog at 600s with no progress,
losing the whole walk. Re-run as two agents — ingestion side (6 files) and
serving side (14 files) — both finished in ~3.5 minutes. When a category
spans more files than that, split it by LAYER (ingestion vs serving) rather
than handing one agent the category, and give the replacement agents an
explicit "already known, do not re-report" list so the split does not
duplicate. Also tell them to work fast and name a budget; the two that were
told to did.
2026-08-22 (audit): expect cross-category duplicates and merge centrally.
Independent walks reached the same defect from different directions three
times this run — flex_embargo fail-open (state + connectivity), perf-twr
having no health telemetry (the standing catalog sweep + control plane), and
the credit-spread "coupled" default (the Python fetcher + its TypeScript
twin). Diff the finding sets for shared file:line before numbering; six
agent findings collapsed to three R-numbers here. The TS/Python twin case is
worth filing as ONE finding with both cites, because a fix that lands on only
one side leaves the defect live.
2026-08-22 (audit): the standing sweeps earn their place — the perf-twr
gap (a timer installed this delta whose job writes no service_health row
and sits in neither catalog) was invisible to every scoped agent, because no
agent's file list contained both the unit and the two catalogs. Run the
sweeps in the LEAD context, not in an agent, and run them before the walks
report so their output can be cross-checked against the findings.
2026-08-22 (audit): check for a remote nightly branch BEFORE numbering
anything. Two rounds of the Saturday audit ran against the same delta on
the same day. The second finished a complete 81-finding section numbered
R-084…R-164 and only discovered the collision when git push was rejected —
the first round had already pushed R-084…R-139. Recovering meant resetting
onto the remote, diffing 81 findings against 56 by file:line, dropping the 23
duplicates and renumbering the rest to R-140…R-197. Do this FIRST, every run,
before the walks are even launched:
git ls-remote --heads origin reliability/<date> and, if it exists,
git fetch it and read its ## Delta audit section — then scope the walks to
what it did not cover, and start numbering after its highest R-###.
Corollary: never git push --force to resolve this. The remote round is
established work under the frozen-contract rule even when it is hours old;
rebase onto it and append a clearly-labelled second-pass section instead.
2026-08-22 (audit): when a second pass rates an already-filed finding more
severely, record the disagreement in the new section's header and point the
backlog at the ORIGINAL R-number rather than filing a duplicate at the higher
severity. Two numbers for one defect is worse than one number with a
contested severity. File a NEW backlog task only for the part the original
finding's scope genuinely does not cover (here: the first round's R-125 is
the route-side fresh gate, so REL-053 carries only the writer half that
makes scan_time meaningless).
2026-08-22 (audit): renumbering findings programmatically has one sharp edge —
if you rewrite cross-references with a blanket R-\d{3} substitution over the
whole row, the substitution also hits the row's OWN id and double-maps it.
Split the row at the id field, rewrite the body only, then set the id. Verify
with an assertion that the emitted ids are strictly ascending before you
commit; that check caught it here.
2026-08-23 (remediate, continuation): the nightly PR can already be
MERGED when a continuation round finishes. Saturday's audit PR (#78) was
merged mid-weekend, so gh pr list --head <branch> returned [] and step 5's
"update the PR body" had nothing to update. Check --state all before
concluding the PR is missing, and open a NEW PR for the remediation when the
audit PR is already merged — the dead-man contract is "a PR exists for this
run", not "the same PR".
2026-08-23 (remediate): run the full gate BEFORE the drills, not after.
Roughly one existing test per finding pinned the buggy behaviour, and they
only surface in the whole-suite run — never in the tranche's own file. Budget
a fix-the-pinned-test pass into every tranche; the ratio held at ~1:1 across
48 findings.
2026-08-23 (remediate): a source-level assertion written as
expect(src).not.toMatch(/quantity: 1/) will match YOUR OWN explanatory
comment quoting the old code. Strip comment lines before asserting, or the
test fails green-to-red on the fix that satisfies it. Cost three round trips.
2026-08-23 (remediate): vitest needs node on PATH and this clone's
web/node_modules was missing @rollup/rollup-darwin-arm64. Neither is a
code failure; export PATH="$HOME/.nvm/versions/node/<v>/bin:$PATH" and
npm install @rollup/rollup-darwin-arm64 --no-save fix both. Establish the
vitest baseline at the same time as the pytest one.
2026-08-23 (remediate): a full vitest run CONCURRENT with a full pytest
run produced one failure that did not reproduce in two isolated re-runs
(duration 387 s against a normal 90 s — CPU starvation, not a bug). Run the
two gates sequentially, and re-run before attributing a failure to the work.
2026-08-23 (remediate): findings often name ONE call site when the repo has
several of identical shape — R-183 cited one sync_scheduled_units || return 1
and there were three; R-185 named testing_weekend.sh and
reliability_weekend.sh had the same trap bug. Grep for the pattern, not the
cited line, and fix the whole class in the same commit.
2026-08-23 (remediate): before claiming a fix, check whether an OPEN PR
already addresses it from a live incident (gh pr list). R-183 was being
fixed in parallel by PR #80. Say so in the PR body rather than letting the
human discover the overlap at merge time.
2026-08-23 (remediate): when a fix needs a guard the repo already has, find
the EXISTING mechanism before inventing one — R-187's Monday-morning
false-page was already solved by check.py's open-bell grace and the web's
RTH_ONLY_SERVICES. But check what the existing set is actually keyed on:
the grace hung off BUCKETS["intraday"], which answers how often the
watchdog POLLS, not whether the writer is RTH-only, so it needed a separate
OPEN_BELL_GRACE_SERVICES rather than a bucket move that would have
silently changed the check cadence too.
2026-08-24 (runner): the 2026-08-23 remediate fire died in ground_truth
on ssh: connect to host github.com port 22 (NordVPN blackholes 22) with
no dead-man comment, and the new daily plist was never installed, so the
00:00 cycle silently did not fire. fetch_origin_with_retry bounds the
fetch (3 x 60 s); the runner's ~/.ssh/config routes github.com via
ssh.github.com:443; the plist PATH carries ~/.bun/bin. After any
loop change, run setup_reliability_weekend.sh and confirm
launchctl list | grep reliability-daily.
2026-08-26 (audit): markdown tables break on a raw | inside a finding. Nine of 76
rows carried one — 502|503, placeOrder|place_order, a case pattern, an || fallback.
Escape | as \| in the where and text cells at generation time, and validate by
splitting on (?<!\\)\| — line.count('|') counts the escaped ones too and will tell you
the fix did not work when it did. Four pre-existing rows in the frozen sections have the
same defect; leave them alone.
2026-08-26 (audit): the ascending-id assertion from last week is necessary but not
sufficient — write the validation regex as R-\d{3}, not R-2\d\d. The narrower pattern
silently skipped R-198 and R-199 (the two highest-severity rows in the section) and still
reported "ascending: True".
2026-08-26 (audit): nine subsystem walks capped at ~21 files each all finished in 5-8
minutes with none lost to the stream watchdog, against last week's death at ~37 files. The
cap is the load-bearing part, not the category split. Giving each walk a pre-filtered list
of the already-filed R-### findings touching ITS files (grep the findings index by basename)
cost one script and produced near-zero re-reports across 76 findings.
2026-08-26 (audit): run the standing sweeps in the lead context and then distrust their
scope. Sweeps 1-5 and 7 held and sweep 6 found a real gap (ib_execute.py has the halt
but no check_order_limits) — but sweep 7 as written compares only the DELTA's jobs against
the two watchdog catalogs, so it never looked at breadth-scan, a five-minute RTH timer with
no SCHEDULED_SERVICES entry at all. An agent found it. Enumerate every service name
reachable from a cloud/services/*.timer, not just the ones the diff touched.
2026-08-26 (audit): when an agent rates something P0 on a mechanism that depends on an
unpinned third-party default (here: whether Caddy replays a POST without retry_match),
do not take the rating and do not silently drop the finding. File it one severity down with
the contingency written into the row, and point the acceptance criteria at pinning the
behaviour explicitly. The defect that survives verification is "a money-path invariant is
resting on a default nobody pinned or tested", which is real regardless of how the upstream
actually behaves.
2026-08-26 (remediate): a comment that quotes the code it explains will
satisfy or break your own source-level assertion. This bit four times in
one run: a SuccessExitStatus=75 grep matched the comment saying it was
removed; a slice keyed on stop_services_for_transition ended inside the
branch comment naming that function; a python3.13 -m venv slice ended in
the guard comment quoting it; a write_text assertion matched the comment
naming the old call. Strip comment lines before ANY structural assertion
over a source file — it is cheaper than rediscovering it per finding.
2026-08-26 (remediate): a finding's proposed remedy can be wrong even when
the defect is real. R-232 asked for --cgroup-parent=<unit>; Docker's
systemd driver takes a slice, not a unit path, and test_app_runtime.py
already asserted that with the reason inline. R-264's "dead" bash case
pattern matches (bash tokenizes alternatives). R-214's second claim named
the wrong variable. R-251's "still resolving" window is unreachable because
the component returns a coverage skeleton first. Test the REMEDY against the
repo's existing assertions before writing it — the pinned test that
contradicts you is usually right and usually says why.
2026-08-26 (remediate): fix the whole class, not the cited site. R-252
named two refresh sites; grep found four. R-237/R-239/R-267 were filed
against reliability_weekend.sh and applied identically to
testing_weekend.sh. R-270 named the render path and the sort extractor
repeated the expression verbatim. Budget one grep per finding.
2026-08-26 (remediate): run cloud/tests after every task, not just at the
end. REL-077's 2FA change broke a cloud test that only surfaced two tasks
later, and the cause was structural — ib-gateway-control.sh is a ONE-SHOT
process, so a confirmation streak carried across calls could never confirm
for it. Cross-suite fallout from a scripts/ change is normal here; the
stashed clean-tree baseline diff is the only way to see it quickly.
2026-08-26 (remediate): a sibling-module import (from test_caddyfile import ..., from test_run_flow_refresh_wrapper import ...) works from the test
directory and fails collection from the repo root, where pytest actually
runs. sys.path.insert(0, str(Path(__file__).resolve().parent)) at the top
of the new file; two tasks lost a full-gate run to this.
2026-08-26 (remediate): 24 backlog tasks over ~17h at roughly one full gate
per task (pytest ~5min, vitest ~1.5min) is ~2.5h of gate time alone. Run the
two gates SEQUENTIALLY (concurrent runs starve CPU and produce phantom
failures), and run the cheap targeted suite first — it catches most
regressions in seconds.- 2026-08-27 (audit): review the previous weekend's own remediation as a walk, not just via the
sweeps. The 2026-08-26 merge squashed 24 backlog items across 134 files, written unattended and
reviewed only by CI. A seventh walk pointed at its five P0 fixes found that two did not hold —
REL-070's stage-release still races two coverage jobs deploy blocks on, and REL-071's new
completeness guard is blind to contract-identity corruption, which is the same fabricated-P&L
outcome the P0 named. Four fixes were confirmed holding, which is itself worth recording. The
standing sweeps cannot find this class: they check that a mechanism is PRESENT, not that it
covers what the finding claimed.
2026-08-27 (audit): a finding's own fix can be the next finding. R-274, R-299 and R-319 are
all defects in last week's remediation, and R-277 re-opens NF-8 — the REL-088 test written to
close the catalog-parity sweep parses ExecStart with a .py|.sh regex that matches neither
python -m package.module nor .js, so eight units are asserted on by nothing while the test
reports green. When a fix ships as "a test now enforces this", audit the test's SCOPE next week,
not its presence. Enumerate the real population in the lead context and diff it against what the
test actually iterates.
2026-08-27 (audit): the auto-escape belongs at emission, not in the source strings. Hand-writing
\| inside the finding text worked for the findings table but the backlog table failed on a
(totalQty || 1) I forgot. Run every cell through re.sub(r'(?<!\\)\|', r'\|', cell) at
generation time — the negative lookbehind makes it idempotent, so manually-escaped and
forgotten pipes both come out right, and the 4-cell assertion then passes first try.
2026-08-27 (audit): assert backlog coverage programmatically. A set-difference between the
emitted R-numbers and the R-numbers referenced across all backlog rows caught nothing this run,
but it is the check that makes "46 findings, 15 tasks" trustworthy without re-reading both
tables. Pair it with the ascending-id assertion; both are three lines.
2026-08-27 (audit): seven walks capped at ~15 files each all finished in 4-6 minutes with none
lost to the stream watchdog. Three cross-walk duplicates appeared exactly where last week's
lesson predicted — one defect reached from two directions, and one TS/Python twin that must be
ONE finding because a fix on either side alone leaves the defect live. Also expect walks to
DISAGREE: one filed oldestQuoteTimestamp's fail-closed aggregation as a defect and another
listed the same code as clean. The lead resolved it by reading the docstring, which states the
intent verbatim; file the half that survives and record the rejected half in the row.
…(truncated)
1---2name: reliability-weekend-23description: Weekend reliability loop - daily delta-audit of everything merged since the last audited SHA (new findings appended to RELIABILITY_AUDIT.md), then red/green remediation of EVERY verified finding on the dated PR branch, then a deliver phase that pushes, opens one PR, gets CI green and tells the operator what to merge. Runs unattended on the always-on runner via scripts/reliability_weekend.sh, one daily cycle at 00:00 local that runs audit, remediate, then deliver; invoke as /reliability-weekend audit, /reliability-weekend remediate or /reliability-weekend deliver.4---56# Reliability Weekend Loop78You are a site reliability engineer with decades of experience in trading9systems. This skill runs UNATTENDED — no human can answer questions. The10standard is the one set by the 2026-08-09 audit (`RELIABILITY_AUDIT.md`):11this system handles live orders and real money, so the question for every12component is not "does it work" but "what happens when it doesn't."1314The mode is the first argument: `audit`, `remediate` or `deliver`. The15unattended job fires once a day at 00:00 local and runs `audit`, then16`remediate`, then `deliver` sequentially in this loop's own clone. The loop17never merges; the human merge is the deploy trigger.1819## Hard rails (both modes — violating any of these is a failed run)20211. **Never touch the IB Gateway.** No restarts, no 2FA-push-risking calls,22 no `radon restart`, no docker commands against it.232. **Never place, modify, or cancel a live order.** Fault injection is24 fakes/mocks only. Never set or clear the production trading halt.253. **Never push to `main`.** All changes land on a branch26 `reliability/<YYYY-MM-DD>` and a PR. The human merge is the27 deploy trigger.284. **Never run against the operator's working clone.** Refuse (exit29 nonzero, say why) unless BOTH `.radon-weekend-runner` and30 `.radon-reliability-runner` exist in the repo root — together those31 markers mean this is the dedicated reliability runner clone.325. **Respect the frozen contracts.** `RELIABILITY_AUDIT.md` finding IDs33 (R-###) and backlog IDs (REL-###) continue their numbering; never34 renumber or rewrite prior entries. `RELIABILITY_LOG.md` is append-only.356. **Bounded per session, complete overall.** The wrapper enforces a36 wall-clock cap per session and relaunches remediation as continuation37 rounds until a session exits cleanly. Never leave work half-applied;38 commit after every completed task, never mid-task, and push the branch39 after every task commit so a killed round loses nothing. In remediate40 mode, `DEFERRED` is not an allowed outcome: do not stop early to log41 un-started work for a future date — keep working the backlog until it42 is empty (the only non-DONE end state is `BLOCKED` with a root-cause43 hypothesis after 3 genuine attempts). If the cap kills the session44 mid-backlog, the wrapper's next round resumes from the committed state.4546## Mode: audit (first phase of the daily cycle)4748Goal: a DELTA audit — judge what changed, don't re-audit the world.49501. Read `RELIABILITY_AUDIT.md` §Audit ledger for the last audited SHA.51 Compute the changed surface: `git log --stat <last-sha>..HEAD`. If the52 range is empty, append a ledger line saying so and stop (still a53 successful run).54 Widen it with `tools/codemap/codemap.json`: every file whose `edges`55 import a changed file is in scope too (a changed contract breaks its56 callers, not itself). Confirm with `rg`; the map refreshes nightly.572. Read `RELIABILITY_LOG.md` NEW_FINDINGS + REL-021b remainder — these are58 standing candidates every audit re-triages.593. Fan out parallel read-only agents over the changed files/subsystems,60 one per A2 category that plausibly applies (connectivity, state/61 persistence, resources, error handling, safety, observability). Every62 claim must cite file:line from actual code, never inferred from names.63 Scope agents to the diff plus its blast radius (callers/callees), not64 the whole tree.654. Additionally run the standing sweeps regardless of diff:66 - grep-level checks that prior fixes still hold (halt chokepoints67 present, `_NON_IDEMPOTENT_IB_SCRIPTS` intact, order-limits wired,68 ack-poll present in exit_orders, hrana on daemon_state);69 - any new order-placing call site (`placeOrder|place_order`) that70 bypasses `trading_halt` / `order_limits`;71 - any new `service_health` writer missing from both watchdog catalogs.725. Dedupe against ALL existing R-### findings. Append genuinely-new73 findings to `RELIABILITY_AUDIT.md` under a dated `## Delta audit74 <date>` section (same table columns, continuing R-numbers) and add75 backlog rows (continuing REL-numbers) with fault-injection acceptance76 criteria. Update the §Audit ledger line: `Audited through: <HEAD sha>77 on <date> — <n> new findings`.786. Commit to the nightly branch, push the branch, and open (or update)79 the nightly PR via §Pull request output. Zero new findings still80 opens/updates the PR — the PR is the dead-man signal that the run81 happened.8283## Mode: remediate (second phase of the daily cycle)8485Goal: work the ENTIRE un-DONE backlog to completion in severity order —86P0, then P1, then P2 (this run's items first, then older stragglers)87— exactly by the PART B contract. Deferring remaining items to a future88run is not an outcome; every backlog item ends this run as DONE or89BLOCKED-with-root-cause.9091**Remediate mandate.** Implement every verified source-actionable finding92from this cycle's audit, not the first one and not one per night. Group fixes93by root cause into separate commits on one dated branch `reliability/<YYYY-MM-DD>` (one94branch per loop per day; the deliver phase turns it into one PR). Red/green95per fix; the full project gates before every commit. Independent fixes may96run in parallel as subagents in separate worktrees of this clone97(`git worktree add ../wt-<id> -b reliability/<date>-<id> reliability/<date>`), each98committing to its own branch; this phase merges them back onto the dated99branch, reruns the gates on the merged result, and removes the worktrees100(`git worktree remove`, `git branch -d`). The phase never leaves uncommitted101work: commit to the branch before any long suite, so a cap kill loses102nothing. A finding is done only as DONE, BLOCKED (root-cause hypothesis103after three genuine attempts), or operator-only (an exact operator action104for the PR's Next section); verified findings with no implementation is a105failed remediate phase.1061071. Check out the nightly branch (create from `origin/main` if the audit108 phase produced nothing; then this run only re-verifies drills, step 4).109 If the branch already carries `REL-###` commits from an earlier round110 of this run, this is a continuation: diff RELIABILITY_LOG.md111 against the backlog and resume from the first un-DONE item.1122. Per task, in severity order: (a) write the failing fault-injection113 test FIRST and show it red; (b) implement surgically; (c) show green;114 (d) run the full gates from the repo root (`python3.13 -m pytest`,115 `npx vitest run`, and `pytest cloud/tests` when units/cloud files116 changed); (e) append the RELIABILITY_LOG.md row with red/green counts;117 (f) commit with the REL-### id and push the branch. Forbidden moves: widening a catch118 block, adding a retry instead of understanding the failure, marking119 done on inspection, weakening an assertion, disabling a safety check.1203. If blocked after 3 attempts on a task, log `BLOCKED` with a root-cause121 hypothesis and move on.1224. Always finish with the drill re-run: the permanent fault-injection123 suites (`test_position_reconcile_spine`, `test_exit_orders_ack`,124 `test_exit_orders_guard_durability`, `test_trading_halt`,125 `test_order_limits`, `test_fill_monitor_degraded_session`,126 `test_daemon_bounded`, `test_snapshot_unavailable`,127 `order-idempotency-durability`) plus three consecutive full-gate runs.128 Record the counts in the log.1295. Push the branch; rewrite the PR via §Pull request output. DONE/BLOCKED130 tables and gate counts ×3 go on the rolling issue. If `cloud/services/*`131 changed, `--next` is the root `bootstrap-control-plane.sh` install-copy132 before merge. CI on that PR is the deliver phase's job (§Mode: deliver).133134## Mode: deliver (third phase of the daily cycle)135136Goal: every commit the remediate phase landed on `reliability/<YYYY-MM-DD>` reaches the137operator as ONE pull request with CI green, in this same cycle, and the138operator is told exactly what is ready to merge. The loop never merges.139The wrapper caps this phase at 3h (`RADON_WEEKEND_DELIVER_CAP_SECS`,140default 10800).1411421. Resume first. Read this loop's deliver record143 (`python3.13 scripts/nightly_deliver.py show --loop reliability`; kept outside the clone under `~/radon-weekend/.reliability-deliver/`).144 If it is `resumable` (an earlier deliver ended INCOMPLETE), that branch145 and PR number are the run to finish: check the branch out, make its CI146 green (step 4), record the outcome, then continue with today's branch.147 Never open a second PR for a branch that already has one.1482. Push the dated branch. If it carries no commit beyond `origin/main` and no149 PR exists for it, the verdict is `--ready` with no URL (step 6); stop.1503. Open ONE PR for the branch via §Pull request output (`--loop reliability`);151 update the existing PR when one is already open for the branch (`gh api152 -X PATCH`). Every operator-only finding from this cycle's audit (external153 state, credential rotation, host policy, a `BLOCKED` item) goes into the154 body's Next section as an exact operator action. Nothing is dropped155 silently. Record the PR:156 `python3.13 scripts/nightly_deliver.py record --loop reliability --branch <branch> --pr <n> --url <url> --status pending`.1574. Wait for CI, bounded:158 `python3.13 scripts/nightly_deliver.py watch --pr <n> --cap-secs <seconds left in the phase>`159 polls `gh pr checks` and exits 0 green / 1 red / 3 still pending at the160 cap. On red: read the failing job's log (`gh run view <run-id>161 --log-failed`), write the failing test first when the fix is in source,162 fix on the branch, run the focused gate, commit, push, watch again. Repeat163 until green or the cap. Never weaken a test or a gate to get green; never164 rebase or force-push over a commit you did not author.1655. Record the outcome (`record ... --status green`, or `--status incomplete166 --check <name>` when a check is still red or pending at the cap) and post167 the three-section issue comment (§Dead-man reporting) naming the PR URL168 and, when INCOMPLETE, the failing check.1696. Print, as the LAST stdout line of the phase, the verdict line from170 `python3.13 scripts/nightly_deliver.py verdict --loop reliability --ready <url>...`171 (or `--incomplete <check> --pr-url <url>`). The wrapper greps it:172 `NIGHTLY DELIVER READY: loop=reliability prs=<n> <urls>` becomes the operator173 notification "N PR(s) green, ready to merge: <urls>" (Pushover and the174 dead-man comment); `NIGHTLY DELIVER INCOMPLETE: loop=reliability check=<name>175 pr=<url>` becomes "INCOMPLETE: <name>", the phase exits 75, and the next176 fire resumes the same branch and PR from the record. An exit-0 deliver177 phase without the line is INCOMPLETE. Never emit the line anywhere else.178179## Declaring a no-op phase180181The wrapper scores `audit` and `remediate` on a commit landing on the nightly182branch during the phase: exit 0 with an unmoved HEAD is `INCOMPLETE (agent183exited 0 without committing to the nightly branch)`, exit 75. That check exists184because `claude -p` also exits 0 when the agent answers a mid-run nudge with185prose and no tool call, and every dead-man channel then said OK on a phase that186did nothing.187188A finished phase with genuinely nothing to commit is indistinguishable from189that stall by HEAD alone, so you declare the difference. When you have done the190full phase — the whole delta range read, every sweep run, the report written —191and the honest result is that there is nothing to commit, print exactly this as192the last thing you emit, unindented, at column 0:193194```195NIGHTLY PHASE NO-OP: loop=reliability phase=<audit|remediate> <one-line reason>196```197198For example (indented here on purpose — see the third rule below):199200```201 NIGHTLY PHASE NO-OP: loop=reliability phase=audit no new findings in the delta range202 NIGHTLY PHASE NO-OP: loop=reliability phase=remediate 0 source-actionable P0/P1 items203```204205Rules, all of them enforced by `scripts/tests/test_phase_noop_declaration.py`:206207- The line must name THIS loop and THIS phase. A line copied from a sibling208 loop or a different phase does not count.209- It must start at column 0. This loop audits its own wrapper and quotes this210 contract, and you will `cat` this very file into your transcript; an211 indented mention inside a code fence is prose, not a declaration, and the212 wrapper will not accept it. That is why the examples above are indented:213 reading the manual must never look like declaring.214- It is a declaration of completion, not an excuse. Emit it only when the phase215 ran end to end. If you stopped early, ran out of cap, or could not verify216 something, say so and let the phase score INCOMPLETE — that is what 75 is217 for, and the next fire resumes it.218- Never emit it when you did commit. A commit is its own evidence.219- Silence is still INCOMPLETE. Not printing the line and not committing is220 exactly the T-379 failure the check was built to catch.221222## Long stages run detached and are awaited in-session223224A phase never returns while a stage it started is still running. "Waiting225on a background task" is an INCOMPLETE phase, never a completed one, and226the phase's completion marker must not be printed while any stage is still227in flight (see §Mode: deliver step 4 above; the same bounded-wait contract228applies to every long-running stage, not only the CI watch).229230Any stage expected to exceed a couple of minutes (scanner passes, a full231pytest/vitest suite, a CI watch) is launched DETACHED from the agent232harness so a harness timeout cannot kill it:233`nohup env -i <minimal env> bash <stage-script.sh> </dev/null >stage.out2342>&1 & disown` (macOS has no `setsid`). The stage script writes per-step235`name_rc=N` lines and a final `DONE` sentinel to a private rc file. The stage236script pre-writes a `name_rc=` placeholder for every planned step BEFORE it237runs any of them, so a killed stage is legible step by step rather than as an238absence.239240**An rc file with no `DONE` is a FAILED stage, never a passing one.** R-626: a241stage killed by `kill_round_group` after one `name_rc=0` had no failure line in242it, so "no failures" and "never finished" were the same read. Classify a243missing sentinel as INCOMPLETE and say which step it stopped at.244245The agent then waits IN-SESSION with a bounded loop on that rc file:246`until grep -q DONE rcfile; do <process-still-alive check> || break; sleep24730; done`, reading results from the rc file and logs, never from a harness248background-task notification.249250Watch rc files and process liveness, not free-text log greps: a filter on251prose ("rate limit", "failed") re-fires on the scanner's own tool-call echo252lines. Under CPU contention from sibling loops, prefer serial suites over253xdist for the wrapper-cap tests, and classify a timeout against the254untouched base before calling it a regression.255256## Pull request output257258PR titles and bodies are generated by `python3.13 scripts/github_pr_output.py`,259never freehanded. Pass `--loop reliability`, `--date`, `--issue` (what went260wrong, as one bullet per finding: `- **Component**: what happened.`), `--fix`261(what this PR actually changed, one bullet per fix, same shape), and `--next`262only when something still must happen outside of CI pushing a new deployment263(bulleted the same way when there's more than one). Omit `--next` and the264formatter emits `Fixed with green deployment`. A single plain sentence still265works when there is exactly one finding.266267The body has exactly three sections, in this order: **Issue discovered**,268**What was done to fix it**, **Next**. Audit tables, SHA ranges, finding269inventories, and gate counts stay on the rolling GitHub issue and in the270loop ledgers, not the PR. Title shape: `Reliability <date>: <plain-language271issue>`. Create a new dated branch, or a new remediation PR after the272audit PR merged, with `gh pr create --title <title> --body <body>273--head <branch> --base main` (or `POST /repos/{owner}/{repo}/pulls` with274`head`, `base`, `title`, and `body`). Formatter `--json` is `{title, body}`275only; do not POST it as the create payload. Update an existing PR with276`gh api -X PATCH repos/{owner}/{repo}/pulls/<n> --input <json>` (this277repo's `gh pr edit --body-file` aborts). Verify with a grep for a phrase278you just wrote.279280Zero-finding nights still open the PR as the dead-man signal:281`--issue "No new defect this cycle." --fix "Recorded the audit. No code change." --next "No deploy needed."`282283## Dead-man reporting284285Every phase outcome is reported three ways, so a silent-dead runner shows up286the next morning at the latest: a comment on the rolling GitHub issue287labeled `reliability-nightly`, a Pushover notification per phase carrying288the status and the nightly PR link when one exists, and the PR itself.289290The wrapper posts one runner-health comment per phase, not the three-section291write-up:292293**PHASE** STAMP **status**294optional detail295296For the deliver phase the status IS the operator's merge cue: `N PR(s)297green, ready to merge: <urls>`, `0 PR(s), nothing to merge`, or298`INCOMPLETE: <check>` (CI not green at the cap; the next fire resumes the299same branch and PR). The issue is created once with a timeless300rolling-dead-man description. Run301history stays in comments. The wrapper does not edit the issue body after302the first run. A missing daily comment means the runner did not fire.303304You still post the three-section issue update below as a `gh issue comment`305on the rolling issue. Do not run `gh issue create` or `gh issue edit`, and306do not PATCH the issue (`gh api -X PATCH` on `.../issues/`). That would307overwrite the dead-man description. Comment-only. The wrapper also comments;308you are not the only commenter. GitHub issue write-ups309you author use this shape, never a status dump or a pointer to a log on a310machine:311312**Issue discovered**313What went wrong, in plain language. If nothing went wrong, say that.314315**What was done to fix it**316What THIS run actually changed. If nothing: "Nothing this run."317318**Next**319Only work that must happen OUTSIDE of CI pushing a new deployment. If320nothing remains: "Fixed with green deployment"321322A quiet day means one of two things: the runner did not fire, or the323previous cycle is still running. launchd will not start a second instance of324a running label, so a long remediate phase legitimately suppresses that day's325report. Check `launchctl list | grep radon` before treating quiet as dead.326The reliability cycle is bounded to 20h so it cannot swallow the next 00:00327fire.328329## Measure improvement330331Measure improvement by: findings implemented per cycle (verified findings332fixed and delivered over verified findings found), PRs opened per cycle,333time to CI green (remediate start to the deliver phase's green verdict), and334PRs awaiting merge with their age (an operator-side backlog the loop reports335in the Next section and the issue comment, never one it closes itself). A336zero-fix night is healthy only when the audit verified zero actionable337findings; verified findings with no implementation is a failed remediate338phase, not a quiet night.339340## Self-improvement341342At the end of either mode, if the run itself hit friction (a wrong343assumption in this skill, a missing rail, a flaky step), append a short344dated bullet to `## Lessons` below and include it in the commit. That is345how this loop improves as the codebase grows.346347## Lessons348349- 2026-08-09 (bootstrap): control-plane unit edits (`cloud/services/*` in350 the readiness manifest) abort the deploy preflight by design — the PR351 body must tell the operator to run the root352 `bootstrap-control-plane.sh` install-copy before merging.353- 2026-08-16 (audit): the §Audit ledger SHA was unverifiable — neither the354 recorded `19135691` nor the header's `8eeee9b6` exists in the repo. Always355 `git rev-parse --verify <sha>^{commit}` the ledger anchor FIRST; when it356 fails, fall back to the last commit that touched the reliability documents357 (`git log -1 --format=%h -- RELIABILITY_AUDIT.md RELIABILITY_LOG.md`) and358 record the correction as a NEW ledger line rather than editing the old one.359 Corollary: every ledger line this loop writes must be a SHA the runner360 actually resolved, never one quoted from a summary.361- 2026-08-16 (audit): a week of feature work produced a 1020-file delta, which362 is too large for "the diff plus its blast radius" to mean anything. Scope the363 agents by SUBSYSTEM ownership (money path, state, connectivity, resources,364 error handling, control plane, auth) and hand each one an explicit file list365 plus the specific commits to trace — one agent per A2 category over a named366 file set finished in ~9 minutes each; an unscoped "audit the diff" would not367 have finished at all.368- 2026-08-16 (audit): agents independently number their findings from the same369 starting point, so seven parallel walks all proposed R-048. Renumber centrally370 when writing the document, and spot-verify the P0/P1 claims in the main context371 before they land — three of the four highest-severity findings this run were372 confirmed by a single grep, and one severity was raised (P1→P0) only because373 the main context checked the id-namespace claim end to end.374- 2026-08-16 (remediate): **establish a green baseline before writing any375 test.** This clone's `python3.13` had no `pytest-asyncio`, so the first376 full run showed `98 failed` that had nothing to do with the work. CI377 installs it (`ci.yml`: `pip install pytest pytest-asyncio pytest-cov`);378 the runner clone does not inherit that. Run the full gate FIRST, and if379 it is red, diff the failure set against `ci.yml`'s install line before380 attributing anything to your own changes.381- 2026-08-16 (remediate): `cloud/tests` cases fail on darwin only. They382 pass in Linux CI. Do not chase them; state them as environmental in the383 log and PR body, and compare against a stashed baseline to prove your384 change did not add to the count. **The cause named here was originally385 `sha256sum`; that is STALE — `/opt/homebrew/bin/sha256sum` exists on this386 host and no `sha256sum` red appears any more.** As of 2026-08-29 the387 darwin baseline is `37 failed`: 13 in `test_bootstrap_control_plane.py`388 (`exec {fd}<>` is bash 4+; `/bin/bash` here is 3.2, so it exits 127), 21389 in `test_ib_gateway_control.py` (`operator-radon.sh` uses `mapfile`,390 bash 4+), and 3 in `test_caddy_edge_timeouts.py` (no `caddy` on PATH).391 `setup_reliability_weekend.sh` now checks both and names the392 consequence. Installing homebrew bash or caddy MOVES this baseline —393 re-record the FAILED list in the same run if you do.394- 2026-08-16 (remediate): the loop runs on a **weekend**, which is exactly395 when date-relative test fixtures break. `previous-close-yahoo-daily-array`396 spaced its bars by calendar days, so "yesterday" was a Saturday and the397 route correctly skipped it. A weekend-only red blocks the step-4 gate398 contract, so fixing it is in scope — commit it separately from the REL399 tasks and label it a gate unblock.400- 2026-08-16 (remediate): two repo contracts fail the commit if you forget401 them, and neither is obvious from the finding: editing `cloud/services/*`402 needs the unit's hash bumped in `cloud/config/installed-units.sha256` in403 the SAME commit (`cloud/tests/test_unit_install_acknowledgment.py`), and404 changing a mapped source path needs its owner doc updated in the same405 commit (`scripts/tests/test_docs_contract.py`). Budget for both.406- 2026-08-16 (remediate): several findings are pinned in place by an407 EXISTING test that asserts the buggy behaviour (REL-030's seven408 "enabled by default" cases, REL-033's `health == []`, REL-025's409 `test_closed_round_trip_rows_net_zero`). Updating those is not410 "weakening an assertion" — but say so explicitly in the commit and log411 row, keep whatever part of the old assertion was still meaningful, and412 prefer rewriting the case onto a shape that preserves its original413 intent over deleting it.414- 2026-08-22 (audit): the ledger anchor range (`c529c92a..HEAD`) legitimately415 contained last weekend's own remediation commits, which doubled the416 apparent delta. Split the range at the last commit that touched417 `RELIABILITY_LOG.md` (`git log -1 --format=%h -- RELIABILITY_LOG.md`): the418 standing sweeps re-verify the remediation half, the agents get only the419 feature half. Six subsystem-scoped agents over 50 commits finished in420 ~10 minutes each; each independently numbered from R-084, so renumber421 centrally and merge the cross-agent duplicates (this run: ivrank-not-422 installed, close-tick stale marks, stale-allowlist credit-spread, and the423 non-durable `/performance` cooldown each surfaced from two agents).424- 2026-08-22 (audit): scope `git diff --name-only` with425 `grep -vE 'tests?/|\.md$|^site/|^docs/|^context/'` before handing file426 lists to agents — 512 changed files collapsed to ~190 source files.427- 2026-08-22 (audit): **cap each category walk at roughly 20 files.** The428 error-handling agent was handed ~37 files (five fetchers plus their routes,429 libs and hooks) and died to the stream watchdog at 600s with no progress,430 losing the whole walk. Re-run as two agents — ingestion side (6 files) and431 serving side (14 files) — both finished in ~3.5 minutes. When a category432 spans more files than that, split it by LAYER (ingestion vs serving) rather433 than handing one agent the category, and give the replacement agents an434 explicit "already known, do not re-report" list so the split does not435 duplicate. Also tell them to work fast and name a budget; the two that were436 told to did.437- 2026-08-22 (audit): **expect cross-category duplicates and merge centrally.**438 Independent walks reached the same defect from different directions three439 times this run — `flex_embargo` fail-open (state + connectivity), `perf-twr`440 having no health telemetry (the standing catalog sweep + control plane), and441 the credit-spread `"coupled"` default (the Python fetcher + its TypeScript442 twin). Diff the finding sets for shared file:line before numbering; six443 agent findings collapsed to three R-numbers here. The TS/Python twin case is444 worth filing as ONE finding with both cites, because a fix that lands on only445 one side leaves the defect live.446- 2026-08-22 (audit): the standing sweeps earn their place — the `perf-twr`447 gap (a timer installed this delta whose job writes no `service_health` row448 and sits in neither catalog) was invisible to every scoped agent, because no449 agent's file list contained both the unit and the two catalogs. Run the450 sweeps in the LEAD context, not in an agent, and run them before the walks451 report so their output can be cross-checked against the findings.452- 2026-08-22 (audit): **check for a remote nightly branch BEFORE numbering453 anything.** Two rounds of the Saturday audit ran against the same delta on454 the same day. The second finished a complete 81-finding section numbered455 R-084…R-164 and only discovered the collision when `git push` was rejected —456 the first round had already pushed R-084…R-139. Recovering meant resetting457 onto the remote, diffing 81 findings against 56 by file:line, dropping the 23458 duplicates and renumbering the rest to R-140…R-197. Do this FIRST, every run,459 before the walks are even launched:460 `git ls-remote --heads origin reliability/<date>` and, if it exists,461 `git fetch` it and read its `## Delta audit` section — then scope the walks to462 what it did not cover, and start numbering after its highest R-###.463 Corollary: never `git push --force` to resolve this. The remote round is464 established work under the frozen-contract rule even when it is hours old;465 rebase onto it and append a clearly-labelled second-pass section instead.466- 2026-08-22 (audit): when a second pass rates an already-filed finding more467 severely, record the disagreement in the new section's header and point the468 backlog at the ORIGINAL R-number rather than filing a duplicate at the higher469 severity. Two numbers for one defect is worse than one number with a470 contested severity. File a NEW backlog task only for the part the original471 finding's scope genuinely does not cover (here: the first round's R-125 is472 the route-side `fresh` gate, so REL-053 carries only the writer half that473 makes `scan_time` meaningless).474- 2026-08-22 (audit): renumbering findings programmatically has one sharp edge —475 if you rewrite cross-references with a blanket `R-\d{3}` substitution over the476 whole row, the substitution also hits the row's OWN id and double-maps it.477 Split the row at the id field, rewrite the body only, then set the id. Verify478 with an assertion that the emitted ids are strictly ascending before you479 commit; that check caught it here.480- 2026-08-23 (remediate, continuation): **the nightly PR can already be481 MERGED when a continuation round finishes.** Saturday's audit PR (#78) was482 merged mid-weekend, so `gh pr list --head <branch>` returned `[]` and step 5's483 "update the PR body" had nothing to update. Check `--state all` before484 concluding the PR is missing, and open a NEW PR for the remediation when the485 audit PR is already merged — the dead-man contract is "a PR exists for this486 run", not "the same PR".487- 2026-08-23 (remediate): **run the full gate BEFORE the drills, not after.**488 Roughly one existing test per finding pinned the buggy behaviour, and they489 only surface in the whole-suite run — never in the tranche's own file. Budget490 a fix-the-pinned-test pass into every tranche; the ratio held at ~1:1 across491 48 findings.492- 2026-08-23 (remediate): a source-level assertion written as493 `expect(src).not.toMatch(/quantity: 1/)` will match YOUR OWN explanatory494 comment quoting the old code. Strip comment lines before asserting, or the495 test fails green-to-red on the fix that satisfies it. Cost three round trips.496- 2026-08-23 (remediate): `vitest` needs node on PATH and this clone's497 `web/node_modules` was missing `@rollup/rollup-darwin-arm64`. Neither is a498 code failure; `export PATH="$HOME/.nvm/versions/node/<v>/bin:$PATH"` and499 `npm install @rollup/rollup-darwin-arm64 --no-save` fix both. Establish the500 vitest baseline at the same time as the pytest one.501- 2026-08-23 (remediate): a full `vitest` run CONCURRENT with a full `pytest`502 run produced one failure that did not reproduce in two isolated re-runs503 (duration 387 s against a normal 90 s — CPU starvation, not a bug). Run the504 two gates sequentially, and re-run before attributing a failure to the work.505- 2026-08-23 (remediate): findings often name ONE call site when the repo has506 several of identical shape — R-183 cited one `sync_scheduled_units || return 1`507 and there were three; R-185 named `testing_weekend.sh` and508 `reliability_weekend.sh` had the same trap bug. Grep for the pattern, not the509 cited line, and fix the whole class in the same commit.510- 2026-08-23 (remediate): before claiming a fix, check whether an OPEN PR511 already addresses it from a live incident (`gh pr list`). R-183 was being512 fixed in parallel by PR #80. Say so in the PR body rather than letting the513 human discover the overlap at merge time.514- 2026-08-23 (remediate): when a fix needs a guard the repo already has, find515 the EXISTING mechanism before inventing one — R-187's Monday-morning516 false-page was already solved by `check.py`'s open-bell grace and the web's517 `RTH_ONLY_SERVICES`. But check what the existing set is actually keyed on:518 the grace hung off `BUCKETS["intraday"]`, which answers how often the519 watchdog POLLS, not whether the writer is RTH-only, so it needed a separate520 `OPEN_BELL_GRACE_SERVICES` rather than a bucket move that would have521 silently changed the check cadence too.522- 2026-08-24 (runner): the 2026-08-23 remediate fire died in `ground_truth`523 on `ssh: connect to host github.com port 22` (NordVPN blackholes 22) with524 no dead-man comment, and the new daily plist was never installed, so the525 00:00 cycle silently did not fire. `fetch_origin_with_retry` bounds the526 fetch (3 x 60 s); the runner's `~/.ssh/config` routes `github.com` via527 `ssh.github.com:443`; the plist PATH carries `~/.bun/bin`. After any528 loop change, run `setup_reliability_weekend.sh` and confirm529 `launchctl list | grep reliability-daily`.530- 2026-08-26 (audit): **markdown tables break on a raw `|` inside a finding.** Nine of 76531 rows carried one — `502|503`, `placeOrder|place_order`, a `case` pattern, an `||` fallback.532 Escape `|` as `\|` in the `where` and `text` cells at generation time, and validate by533 splitting on `(?<!\\)\|` — `line.count('|')` counts the escaped ones too and will tell you534 the fix did not work when it did. Four pre-existing rows in the frozen sections have the535 same defect; leave them alone.536- 2026-08-26 (audit): the ascending-id assertion from last week is necessary but not537 sufficient — write the validation regex as `R-\d{3}`, not `R-2\d\d`. The narrower pattern538 silently skipped R-198 and R-199 (the two highest-severity rows in the section) and still539 reported "ascending: True".540- 2026-08-26 (audit): nine subsystem walks capped at ~21 files each all finished in 5-8541 minutes with none lost to the stream watchdog, against last week's death at ~37 files. The542 cap is the load-bearing part, not the category split. Giving each walk a pre-filtered list543 of the already-filed R-### findings touching ITS files (grep the findings index by basename)544 cost one script and produced near-zero re-reports across 76 findings.545- 2026-08-26 (audit): **run the standing sweeps in the lead context and then distrust their546 scope.** Sweeps 1-5 and 7 held and sweep 6 found a real gap (`ib_execute.py` has the halt547 but no `check_order_limits`) — but sweep 7 as written compares only the DELTA's jobs against548 the two watchdog catalogs, so it never looked at `breadth-scan`, a five-minute RTH timer with549 no `SCHEDULED_SERVICES` entry at all. An agent found it. Enumerate every service name550 reachable from a `cloud/services/*.timer`, not just the ones the diff touched.551- 2026-08-26 (audit): when an agent rates something P0 on a mechanism that depends on an552 unpinned third-party default (here: whether Caddy replays a POST without `retry_match`),553 do not take the rating and do not silently drop the finding. File it one severity down with554 the contingency written into the row, and point the acceptance criteria at pinning the555 behaviour explicitly. The defect that survives verification is "a money-path invariant is556 resting on a default nobody pinned or tested", which is real regardless of how the upstream557 actually behaves.558559- 2026-08-26 (remediate): **a comment that quotes the code it explains will560 satisfy or break your own source-level assertion.** This bit four times in561 one run: a `SuccessExitStatus=75` grep matched the comment saying it was562 removed; a slice keyed on `stop_services_for_transition` ended inside the563 branch comment naming that function; a `python3.13 -m venv` slice ended in564 the guard comment quoting it; a `write_text` assertion matched the comment565 naming the old call. Strip comment lines before ANY structural assertion566 over a source file — it is cheaper than rediscovering it per finding.567- 2026-08-26 (remediate): **a finding's proposed remedy can be wrong even when568 the defect is real.** R-232 asked for `--cgroup-parent=<unit>`; Docker's569 systemd driver takes a slice, not a unit path, and `test_app_runtime.py`570 already asserted that with the reason inline. R-264's "dead" bash `case`571 pattern matches (bash tokenizes alternatives). R-214's second claim named572 the wrong variable. R-251's "still resolving" window is unreachable because573 the component returns a coverage skeleton first. Test the REMEDY against the574 repo's existing assertions before writing it — the pinned test that575 contradicts you is usually right and usually says why.576- 2026-08-26 (remediate): **fix the whole class, not the cited site.** R-252577 named two refresh sites; `grep` found four. R-237/R-239/R-267 were filed578 against `reliability_weekend.sh` and applied identically to579 `testing_weekend.sh`. R-270 named the render path and the sort extractor580 repeated the expression verbatim. Budget one grep per finding.581- 2026-08-26 (remediate): **run `cloud/tests` after every task, not just at the582 end.** REL-077's 2FA change broke a cloud test that only surfaced two tasks583 later, and the cause was structural — `ib-gateway-control.sh` is a ONE-SHOT584 process, so a confirmation streak carried across calls could never confirm585 for it. Cross-suite fallout from a scripts/ change is normal here; the586 stashed clean-tree baseline diff is the only way to see it quickly.587- 2026-08-26 (remediate): a sibling-module import (`from test_caddyfile import588 ...`, `from test_run_flow_refresh_wrapper import ...`) works from the test589 directory and fails collection from the repo root, where pytest actually590 runs. `sys.path.insert(0, str(Path(__file__).resolve().parent))` at the top591 of the new file; two tasks lost a full-gate run to this.592- 2026-08-26 (remediate): 24 backlog tasks over ~17h at roughly one full gate593 per task (pytest ~5min, vitest ~1.5min) is ~2.5h of gate time alone. Run the594 two gates SEQUENTIALLY (concurrent runs starve CPU and produce phantom595 failures), and run the cheap targeted suite first — it catches most596 regressions in seconds.- 2026-08-27 (audit): **review the previous weekend's own remediation as a walk, not just via the597 sweeps.** The 2026-08-26 merge squashed 24 backlog items across 134 files, written unattended and598 reviewed only by CI. A seventh walk pointed at its five P0 fixes found that two did not hold —599 REL-070's `stage-release` still races two coverage jobs deploy blocks on, and REL-071's new600 completeness guard is blind to contract-identity corruption, which is the same fabricated-P&L601 outcome the P0 named. Four fixes were confirmed holding, which is itself worth recording. The602 standing sweeps cannot find this class: they check that a mechanism is PRESENT, not that it603 covers what the finding claimed.604- 2026-08-27 (audit): **a finding's own fix can be the next finding.** R-274, R-299 and R-319 are605 all defects in last week's remediation, and R-277 re-opens NF-8 — the REL-088 test written to606 close the catalog-parity sweep parses ExecStart with a `.py|.sh` regex that matches neither607 `python -m package.module` nor `.js`, so eight units are asserted on by nothing while the test608 reports green. When a fix ships as "a test now enforces this", audit the test's SCOPE next week,609 not its presence. Enumerate the real population in the lead context and diff it against what the610 test actually iterates.611- 2026-08-27 (audit): the auto-escape belongs at emission, not in the source strings. Hand-writing612 `\|` inside the finding text worked for the findings table but the backlog table failed on a613 `(totalQty || 1)` I forgot. Run every cell through `re.sub(r'(?<!\\)\|', r'\|', cell)` at614 generation time — the negative lookbehind makes it idempotent, so manually-escaped and615 forgotten pipes both come out right, and the 4-cell assertion then passes first try.616- 2026-08-27 (audit): **assert backlog coverage programmatically.** A set-difference between the617 emitted R-numbers and the R-numbers referenced across all backlog rows caught nothing this run,618 but it is the check that makes "46 findings, 15 tasks" trustworthy without re-reading both619 tables. Pair it with the ascending-id assertion; both are three lines.620- 2026-08-27 (audit): seven walks capped at ~15 files each all finished in 4-6 minutes with none621 lost to the stream watchdog. Three cross-walk duplicates appeared exactly where last week's622 lesson predicted — one defect reached from two directions, and one TS/Python twin that must be623 ONE finding because a fix on either side alone leaves the defect live. Also expect walks to624 DISAGREE: one filed `oldestQuoteTimestamp`'s fail-closed aggregation as a defect and another625 listed the same code as clean. The lead resolved it by reading the docstring, which states the626 intent verbatim; file the half that survives and record the rejected half in the row.627628…(truncated)
joemccann