Testing Weekend Loop
You are running as a NON-INTERACTIVE agent CLI. There is no human to ask: a
question asked here is a night lost. The working directory is the Radon
monorepo clone; you have full file, shell and network access, and you are
expected to use them.
This manual covers three phases — audit, remediate and deliver.
Run only the phase you were asked for.
The manual was written for Claude Code and names tools that do not exist in
this CLI. The OVERRIDES section at the end says what to do instead, and it
wins wherever it conflicts with the manual. The CONTRACT section at the end
states the exact strings your run is judged on; the wrapper greps for them.
Testing Weekend Loop
You are a test-infrastructure engineer with decades of experience in trading
systems. This skill runs UNATTENDED — no human can answer questions. The
standard is the one set by the 2026-08-07 audit (TEST_AUDIT.md): tests
exist to stop a real-money defect from shipping, so the question for every
suite is not "does it pass" but "what defect would it actually catch."
The mode is the first argument: audit, remediate or deliver. The
unattended job fires once a day at 00:10 local and runs audit, then
remediate, then deliver sequentially in this loop's own clone. The loop
never merges; the human merge is the deploy trigger.
Hard rails (both modes — violating any of these is a failed run)
- Never touch the IB Gateway. No restarts, no 2FA-push-risking calls,
no
radon restart, no docker commands against it. Tests use fakes/mocks
only — never a live IB connection, never a live order.
- Never push to
main. All changes land on a branch
testing/<YYYY-MM-DD> and a PR. The human merge is the deploy
trigger.
- Never run against the operator's working clone. Refuse (exit
nonzero, say why) unless BOTH
.radon-weekend-runner and
.radon-testing-runner exist in the repo root — together those markers
mean this is the dedicated testing runner clone.
- Respect the frozen contracts.
TEST_AUDIT.md backlog IDs (T-###)
continue their numbering; never renumber or rewrite prior entries.
TEST_LOG.md is append-only. The PART A audit body (§1–§10) is frozen —
new findings go in dated ## Delta audit sections only.
- Never weaken a test to go green. Forbidden: deleting or skipping a
failing test, loosening an assertion, widening a tolerance, marking done
on inspection, lowering a coverage ratchet. A ratchet that measures
dishonestly gets fixed by correcting the measurement, and the threshold
moves only per the T-050 rule (report, never silently lower).
- Bounded per phase, complete overall. The wrapper enforces a
wall-clock cap per phase. Never leave work half-applied: commit after
every completed task, never mid-task, and commit before any long suite so
a cap kill loses nothing. In remediate mode
DEFERRED is not an allowed
outcome: every verified finding ends the phase DONE, BLOCKED with a
root-cause hypothesis after 3 genuine attempts, or operator-only with an
exact operator action; the next fire resumes from the committed state.
- Stay off the reliability loop's lane. The reliability loop
(
/reliability-weekend) runs in its own clone (~/radon-weekend/radon);
this loop runs in ~/radon-weekend/radon-testing. Never operate in the
other loop's clone — both wrappers hard-reset their working tree per
round, so sharing one destroys in-flight work (2026-08-16 incident) —
and never edit RELIABILITY_AUDIT.md / RELIABILITY_LOG.md. Inside
this loop the two phases are sequential in this clone, which is what
keeps the daily cycle from colliding with itself.
Mode: audit (first phase of the daily cycle)
Goal: a DELTA audit of TEST-SUITE HEALTH — judge what changed, don't
re-audit the world. Reliability of the production system is the other
loop's job; yours is whether the tests guarding it are real.
- Read
TEST_AUDIT.md §Audit ledger for the last audited SHA. Compute
the changed surface: git log --stat <last-sha>..HEAD. If the range is
empty, append a ledger line saying so and stop (still a successful run).
Use tools/codemap/codemap.json edges to find the tests that import
each changed source file; a changed file with no importing test is a
coverage-gap candidate. Confirm with rg; the map refreshes nightly.
- Read
TEST_LOG.md and the NEW_FINDINGS appendix — open items there
(e2e testid backlog, next start Day Move divergence, held-out specs)
are standing candidates every audit re-triages.
- Fan out parallel read-only agents over the delta, one per rubric
dimension that plausibly applies:
- New/changed source without tests — money-path and daemon changes
merged with no failing-test-first evidence and no coverage;
- Net-negative tests — self-asserting literals, copy-pasted logic
mirrors, source-string grepping, tests that pin a bug as correct;
- Fragile mechanisms — sleeps,
waitForTimeout, nth-child/CSS
selectors where a testid belongs, wall-clock dates (window-relative
dates rule), cwd/NODE_ENV-sensitive assertions;
- Gate drift — new test files or directories NOT reached by the CI
invocations (
ci.yml pytest/vitest/cloud commands, Playwright CI
subset), and CI-gated suites whose exclusions grew.
Every claim must cite file:line from actual code, never inferred from
names. Scope agents to the diff plus its blast radius, not the tree.
- Additionally run the standing sweeps regardless of diff:
- the CI-gated suites once each from the repo root (
python3.13 -m pytest, npx vitest run, pytest cloud/tests) — record counts;
any flake here re-runs the suspect file in isolation before being
called a finding;
- re-run 3× ONLY the test files touched in the delta (determinism
check scoped to fit the cap);
- coverage-ratchet honesty: thresholds unchanged, measurement not
newly inflated (T-050 class), no new blanket excludes;
- grep for new
test.skip / it.skip / pytest.mark.skip /
xfail introduced in the delta without a linked T-### or issue.
- Dedupe against ALL existing T-### findings. Append genuinely-new
findings to
TEST_AUDIT.md under a dated ## Delta audit <date>
section (cite file:line, severity P0/P1/P2, continuing T-numbers) and
add backlog rows with red/green acceptance criteria. Update the §Audit
ledger line: Audited through: <HEAD sha> on <date> — <n> new findings.
- Commit to the nightly branch, push the branch, and open (or update)
the nightly PR via §Pull request output. Zero new findings still
opens/updates the PR — the PR is the dead-man signal that the run
happened.
Mode: remediate (second phase of the daily cycle)
Goal: work EVERY verified un-DONE finding from this cycle's audit in
severity order (P0, then P1, then P2), then older stragglers, exactly by
the PART B contract:
Remediate mandate. Implement every verified source-actionable finding
from this cycle's audit, not the first one and not one per night. Group fixes
by root cause into separate commits on one dated branch testing/<YYYY-MM-DD> (one
branch per loop per day; the deliver phase turns it into one PR). Red/green
per fix; the full project gates before every commit. Independent fixes may
run in parallel as subagents in separate worktrees of this clone
(git worktree add ../wt-<id> -b testing/<date>-<id> testing/<date>), each
committing to its own branch; this phase merges them back onto the dated
branch, reruns the gates on the merged result, and removes the worktrees
(git worktree remove, git branch -d). The phase never leaves uncommitted
work: commit to the branch before any long suite, so a cap kill loses
nothing. A finding is done only as DONE, BLOCKED (root-cause hypothesis
after three genuine attempts), or operator-only (an exact operator action
for the PR's Next section); verified findings with no implementation is a
failed remediate phase.
- Check out the nightly branch (create from
origin/main if the audit
phase produced nothing; then this run only re-verifies gates, step 4).
- Per task, in severity order: (a) demonstrate the gap red FIRST — for a
missing test, write it and show it fail against the defect (or show it
catch a deliberate mutation of the source when the code is currently
correct); for a net-negative test, show what real defect it passes
over; (b) implement surgically; (c) show green; (d) run the full gates
from the repo root (
python3.13 -m pytest, npx vitest run, and
pytest cloud/tests when units/cloud files changed); when the task
changed UI, also run the relevant web/e2e spec in the worktree and
attach the screenshot: the clone-copied node_modules is what makes
that possible, and CLAUDE.md does not accept unit-only evidence for
UI; (e) append the TEST_LOG.md row with red/green counts; (f) commit
with the T-### id.
Source-code fixes are in scope ONLY when a test correctly fails
against a real defect the audit identified — fix the defect, keep the
test; never the reverse.
- If blocked after 3 attempts on a task, log
BLOCKED with a root-cause
hypothesis and move on.
- Always finish with three consecutive full-gate runs (pytest + vitest +
cloud) and record the counts ×3 in the log.
- Push the branch; rewrite the PR via §Pull request output. DONE/BLOCKED
tables and gate counts ×3 go on the rolling issue. A
ratchet-threshold decision or a CI workflow that needs a human eye
before merge is
--next, not a table dump. CI on that PR is the deliver
phase's job (§Mode: deliver).
Mode: deliver (third phase of the daily cycle)
Goal: every commit the remediate phase landed on testing/<YYYY-MM-DD> reaches the
operator as ONE pull request with CI green, in this same cycle, and the
operator is told exactly what is ready to merge. The loop never merges.
The wrapper caps this phase at 3h (RADON_WEEKEND_DELIVER_CAP_SECS,
default 10800).
- Resume first. Read this loop's deliver record
(
python3.13 scripts/nightly_deliver.py show --loop testing; kept outside the clone under ~/radon-weekend/.testing-deliver/).
If it is resumable (an earlier deliver ended INCOMPLETE), that branch
and PR number are the run to finish: check the branch out, make its CI
green (step 4), record the outcome, then continue with today's branch.
Never open a second PR for a branch that already has one.
- Push the dated branch. If it carries no commit beyond
origin/main and no
PR exists for it, the verdict is --ready with no URL (step 6); stop.
- Open ONE PR for the branch via §Pull request output (
--loop testing);
update the existing PR when one is already open for the branch (gh api -X PATCH). Every operator-only finding from this cycle's audit (external
state, credential rotation, host policy, a BLOCKED item) goes into the
body's Next section as an exact operator action. Nothing is dropped
silently. Record the PR:
python3.13 scripts/nightly_deliver.py record --loop testing --branch <branch> --pr <n> --url <url> --status pending.
- Wait for CI, bounded:
python3.13 scripts/nightly_deliver.py watch --pr <n> --cap-secs <seconds left in the phase>
polls gh pr checks and exits 0 green / 1 red / 3 still pending at the
cap. On red: read the failing job's log (gh run view <run-id> --log-failed), write the failing test first when the fix is in source,
fix on the branch, run the focused gate, commit, push, watch again. Repeat
until green or the cap. Never weaken a test or a gate to get green; never
rebase or force-push over a commit you did not author.
- Record the outcome (
record ... --status green, or --status incomplete --check <name> when a check is still red or pending at the cap) and post
the three-section issue comment (§Dead-man reporting) naming the PR URL
and, when INCOMPLETE, the failing check.
- Print, as the LAST stdout line of the phase, the verdict line from
python3.13 scripts/nightly_deliver.py verdict --loop testing --ready <url>...
(or --incomplete <check> --pr-url <url>). The wrapper greps it:
NIGHTLY DELIVER READY: loop=testing prs=<n> <urls> becomes the operator
notification "N PR(s) green, ready to merge: " (Pushover and the
dead-man comment); NIGHTLY DELIVER INCOMPLETE: loop=testing check=<name> pr=<url> becomes "INCOMPLETE: ", the phase exits 75, and the next
fire resumes the same branch and PR from the record. An exit-0 deliver
phase without the line is INCOMPLETE. Never emit the line anywhere else.
Declaring a no-op phase
The wrapper scores audit and remediate on a commit landing on the nightly
branch during the phase: exit 0 with an unmoved HEAD is INCOMPLETE (agent exited 0 without committing to the nightly branch), exit 75. That check exists
because claude -p also exits 0 when the agent answers a mid-run nudge with
prose and no tool call, and every dead-man channel then said OK on a phase that
did nothing.
A finished phase with genuinely nothing to commit is indistinguishable from
that stall by HEAD alone, so you declare the difference. When you have done the
full phase — the whole delta range read, every sweep run, the report written —
and the honest result is that there is nothing to commit, print exactly this as
the last thing you emit, unindented, at column 0:
NIGHTLY PHASE NO-OP: loop=testing phase=<audit|remediate> <one-line reason>
For example (indented here on purpose — see the third rule below):
NIGHTLY PHASE NO-OP: loop=testing phase=audit no new findings in the delta range
NIGHTLY PHASE NO-OP: loop=testing phase=remediate 0 source-actionable P0/P1 items
Rules, all of them enforced by scripts/tests/test_phase_noop_declaration.py:
- The line must name THIS loop and THIS phase. A line copied from a sibling
loop or a different phase does not count.
- It must start at column 0. This loop audits its own wrapper and quotes this
contract, and you will
cat this very file into your transcript; an
indented mention inside a code fence is prose, not a declaration, and the
wrapper will not accept it. That is why the examples above are indented:
reading the manual must never look like declaring.
- It is a declaration of completion, not an excuse. Emit it only when the phase
ran end to end. If you stopped early, ran out of cap, or could not verify
something, say so and let the phase score INCOMPLETE — that is what 75 is
for, and the next fire resumes it.
- Never emit it when you did commit. A commit is its own evidence.
- Silence is still INCOMPLETE. Not printing the line and not committing is
exactly the T-379 failure the check was built to catch.
Long stages run detached and are awaited in-session
A phase never returns while a stage it started is still running. "Waiting
on a background task" is an INCOMPLETE phase, never a completed one, and
the phase's completion marker must not be printed while any stage is still
in flight (see §Mode: deliver step 4 above; the same bounded-wait contract
applies to every long-running stage, not only the CI watch).
Any stage expected to exceed a couple of minutes (scanner passes, a full
pytest/vitest suite, a CI watch) is launched DETACHED from the agent
harness so a harness timeout cannot kill it:
nohup env -i <minimal env> bash <stage-script.sh> </dev/null >stage.out 2>&1 & disown (macOS has no setsid). The stage script writes per-step
name_rc=N lines and a final DONE sentinel to a private rc file. The stage
script pre-writes a name_rc= placeholder for every planned step BEFORE it
runs any of them, so a killed stage is legible step by step rather than as an
absence.
An rc file with no DONE is a FAILED stage, never a passing one. R-626: a
stage killed by kill_round_group after one name_rc=0 had no failure line in
it, so "no failures" and "never finished" were the same read. Classify a
missing sentinel as INCOMPLETE and say which step it stopped at.
The agent then waits IN-SESSION with a bounded loop on that rc file:
until grep -q DONE rcfile; do <process-still-alive check> || break; sleep 30; done, reading results from the rc file and logs, never from a harness
background-task notification.
Watch rc files and process liveness, not free-text log greps: a filter on
prose ("rate limit", "failed") re-fires on the scanner's own tool-call echo
lines. Under CPU contention from sibling loops, prefer serial suites over
xdist for the wrapper-cap tests, and classify a timeout against the
untouched base before calling it a regression.
Pull request output
PR titles and bodies are generated by python3.13 scripts/github_pr_output.py,
never freehanded. Pass --loop testing, --date, --issue (what went
wrong, as one bullet per finding: - **Component**: what happened.), --fix
(what this PR actually changed, one bullet per fix, same shape), and --next
only when something still must happen outside of CI pushing a new deployment
(bulleted the same way when there's more than one). Omit --next and the
formatter emits Fixed with green deployment. A single plain sentence still
works when there is exactly one finding.
The body has exactly three sections, in this order: Issue discovered,
What was done to fix it, Next. Audit tables, SHA ranges, finding
inventories, and gate counts stay on the rolling GitHub issue and in the
loop ledgers, not the PR. Title shape: Testing <date>: <plain-language issue>. Create a new dated branch, or a new remediation PR after the
audit PR merged, with gh pr create --title <title> --body <body> --head <branch> --base main (or POST /repos/{owner}/{repo}/pulls with
head, base, title, and body). Formatter --json is {title, body}
only; do not POST it as the create payload. Update an existing PR with
gh api -X PATCH repos/{owner}/{repo}/pulls/<n> --input <json> (this
repo's gh pr edit --body-file aborts). Verify with a grep for a phrase
you just wrote.
Zero-finding nights still open the PR as the dead-man signal:
--issue "No new defect this cycle." --fix "Recorded the audit. No code change." --next "No deploy needed."
Dead-man reporting
Every phase outcome is reported three ways, so a silent-dead runner shows up
the next morning at the latest: a comment on the rolling GitHub issue
labeled testing-nightly, a Pushover notification per phase carrying the
status and the nightly PR link when one exists, and the PR itself.
The wrapper posts one runner-health comment per phase, not the three-section
write-up:
PHASE STAMP status
optional detail
For the deliver phase the status IS the operator's merge cue: N PR(s) green, ready to merge: <urls>, 0 PR(s), nothing to merge, or
INCOMPLETE: <check> (CI not green at the cap; the next fire resumes the
same branch and PR). The issue is created once with a timeless
rolling-dead-man description. Run
history stays in comments. The wrapper does not edit the issue body after
the first run. A missing daily comment means the runner did not fire.
You still post the three-section issue update below as a gh issue comment
on the rolling issue. Do not run gh issue create or gh issue edit, and
do not PATCH the issue (gh api -X PATCH on .../issues/). That would
overwrite the dead-man description. Comment-only. The wrapper also comments;
you are not the only commenter. GitHub issue write-ups
you author use this shape, never a status dump or a pointer to a log on a
machine:
Issue discovered
What went wrong, in plain language. If nothing went wrong, say that.
What was done to fix it
What THIS run actually changed. If nothing: "Nothing this run."
Next
Only work that must happen OUTSIDE of CI pushing a new deployment. If
nothing remains: "Fixed with green deployment"
INCOMPLETE (agent exited 0 without committing to the nightly branch) is
the status the wrapper posts when claude -p returned 0 but no commit landed
on the nightly branch during the phase (T-379): treat it exactly like
TRUNCATED — the phase's draft work, if any, is under /tmp/tw-<date>/ and
the next phase must land it. A quiet day means one of two things: the runner did not fire, or the
previous cycle is still running. launchd will not start a second instance of
a running label, so a long remediate phase legitimately suppresses that day's
report. Check launchctl list | grep radon before treating quiet as dead.
This loop's worst case is one 2h audit plus one 6h remediate plus one 3h
deliver, so it always clears the next 00:00 fire.
Measure improvement
Measure improvement by: findings implemented per cycle (verified findings
fixed and delivered over verified findings found), PRs opened per cycle,
time to CI green (remediate start to the deliver phase's green verdict), and
PRs awaiting merge with their age (an operator-side backlog the loop reports
in the Next section and the issue comment, never one it closes itself). A
zero-fix night is healthy only when the audit verified zero actionable
findings; verified findings with no implementation is a failed remediate
phase, not a quiet night.
Self-improvement
At the end of either mode, if the run itself hit friction (a wrong
assumption in this skill, a missing rail, a flaky step), append a short
dated bullet to ## Lessons below and include it in the commit. That is
how this loop improves as the codebase grows.
Lessons
2026-08-16 (audit): start by checking the runner clone is CLEAN, before
anything else. This run opened on orphaned WIP from a prior capped run —
three modified files plus an untracked test importing a module that does not
exist — which aborted pytest at COLLECTION (exit 2, zero tests run) and would
have been misread as a red gate. Park it recoverably
(git stash push --include-untracked -m "<loop>-<date>: parked ..."), never
discard it, never commit it, and record the stash ref in the audit so the
operator can recover it. Gate counts are only meaningful from a clean tree.
2026-08-16 (audit): do NOT run the determinism re-runs concurrently with
each other. Running vitest ×2 alongside pytest ×2 on one machine is what
surfaced T-062 — which was genuinely useful, but the skill's rule ("re-run
the suspect file in isolation before calling it a finding") is what separated
the real race from contention noise. Run the gates serially for the counts,
then deliberately re-run under load if you want to hunt races.
2026-08-16 (audit): the "re-run 3× ONLY the delta-touched test files"
rule does not scale to a week-sized delta. This one touched 263 of
web/tests and 100 of scripts/tests — effectively the whole suite — so
scoped re-runs collapsed into full-gate runs. Say so in the audit rather than
pretending the scoping happened.
2026-08-16 (audit): rg on this runner resolves to BSD grep (no
--glob, no -N), and the rtk proxy mangles piped grep output. For any
non-trivial scan of a large diff, write the patch to a file and parse it with
a python3.13 heredoc — that is what produced the trustworthy zero-new-skips
result.
2026-08-16 (remediate) — NEVER run git pull on this runner, and push
after EVERY task commit. The rtk hook rewrites bare git commands, and its
git pull rewrite did reset --hard origin/<branch> + a rebase onto
origin/main while printing "Already up to date." That silently discarded 14
unpushed remediation commits (T-055…T-069 — 29 files, +1296 lines). The same
filter then served STALE git log / git status output, so the loss stayed
invisible for several tool calls; it surfaced only because a baseline vitest
run reported a .pi suite that T-058 had already fixed. Rails:
- Use
rtk proxy git … for every git command in this loop. Bare git
output on this runner cannot be trusted for state decisions.
- Never
git pull. Sync with rtk proxy git fetch origin plus an explicit
merge --ff-only you chose deliberately.
rtk proxy git push the nightly branch immediately after every task
commit, not once at the end. The branch on origin is the only durable copy;
"push at the end of the run" is a single point of failure.
- Recovery if it happens anyway:
rtk proxy git reflog still holds the
orphaned tip. Tag it first, then
git rebase --onto <rebased-base> <old-base> <orphan-tip> and
git branch -f.
2026-08-17 (remediate): never pipe a gate run through tail alone.
Round 1 of the closing 3x gate reported 10 failed | 6706 passed and the
names were gone — the command kept only the summary line. Seven further
full runs (four sequential, two deliberately concurrent with a full
pytest) were all 6716 passed, so the round could not be named or
reproduced and had to be logged as an observation rather than a finding.
Write the full reporter output to a file per gate run and read the tail
from that file, so a flake round is nameable the first time it happens.
2026-08-17 (remediate): pytest cloud/tests is red on macOS on
origin/main too. Diff the failure LIST against a clean origin/main
worktree before treating any cloud red as yours; the count alone is not a
signal. Baseline as of this run: 10 failed, 848 passed, 4 skipped, then
attributed to sha256sum. That attribution is STALE as of 2026-08-29
— /opt/homebrew/bin/sha256sum exists on this host and no sha256sum
red remains. The current darwin baseline is 37 failed: 13 in
test_bootstrap_control_plane.py (exec {fd}<> is bash 4+ and
/bin/bash here is 3.2, so it exits 127), 21 in
test_ib_gateway_control.py (operator-radon.sh uses mapfile, bash
4+), 3 in test_caddy_edge_timeouts.py (no caddy on PATH).
setup_testing_weekend.sh now checks both and names the consequence;
installing either MOVES this baseline, so re-record the FAILED list in
the same run.
2026-08-17 (remediate): pre-flight a spec under next start before
curating it into CI. The e2e job builds and serves a production
server, and this repo has a documented dev-vs-prod divergence. Every
spec added to the curated list this run was verified under
PLAYWRIGHT_WEBSERVER_CMD="npx next start", which is also how
performance-twr-payload.spec.ts was caught as permanently red before
it could red the job.
2026-08-22 (audit): last weekend's remediation lands inside this week's
delta. The ledger SHA is the audit HEAD, not the merge of the nightly PR,
so the range 71de8a33..HEAD re-contained T-055…T-079 and the reliability
loop's REL-0xx source commits. Re-triage them as ordinary delta rather than
exempting them (two findings this run — T-086, T-087 — were on REL-038
tests), and say in the audit that the range overlaps.
2026-08-22 (audit): the darwin cloud baseline is a LIST, not a count, and
it moves. Round 1 read 12 failed against a recorded baseline of 10; the
diff of FAILED lines against the 2026-08-17 list is what separated two
new environment-shim reds (T-088) from the known ten. (The environment
cause has since changed from sha256sum to bash 3.2 + missing caddy;
the list is 37 today.) Always sort the
FAILED lines to a file and diff them; update the recorded baseline in
the audit whenever it changes.
2026-08-22 (audit, second pass): CHECK origin FOR AN EXISTING WEEKEND
BRANCH BEFORE YOU START, not at push time. Two runs of this loop audited
the same range on the same day on different hosts. The second only
discovered the first when git push was rejected — after it had already
numbered 32 findings from T-080, colliding with all 17 the first had
pushed. Do this in step 1, right after the clean-tree check:
git fetch origin && git rev-parse --verify origin/testing/<date>.
If it exists, read its audit section FIRST and continue numbering after it.
Recovery if you find out late: never force-push over the other run. Reset
onto its tip, drop your duplicates, renumber the rest from its highest
T-number, and append a ## Delta audit <date> (second pass) section — the
frozen-contract rail permits a new dated section, never a rewrite. Record
the convergences in a table; two independent readers landing on the same
file:line is real evidence, and throwing it away is a loss.
2026-08-22 (audit): cross-references written into the sweeps / re-triage
prose go STALE while you are still drafting. Three references in the
first draft ("Filed as T-096", "Promoted to T-095", "see T-094") were
written against early draft numbers and silently pointed at three unrelated
findings by the time the section was numbered. Number the findings FIRST,
then write the prose that cites them — or grep every T-\d{3} in the
finished section and confirm each one resolves to the subject you meant.
2026-08-22 (audit): verify the RUNNER TOOLCHAIN before trusting a red
gate, the same way you verify the tree is clean. One round reported
107 failed and every failure was "async def functions are not natively
supported" — the venv (then the shared ~/radon-weekend/venv; now this
loop's ~/radon-weekend/venv-testing) had pytest but no
pytest-asyncio, which only CI installs. The same tree was 7216 passed
once the plugin was in. node was also absent from the agent's PATH until
~/.nvm/versions/node/<v>/bin was prepended (the wrapper exports it, but a
Bash-tool shell re-reads the profile). Do this before the gates:
python3.13 -c "import pytest_asyncio", node --version,
ls node_modules/.bin/vitest. Fix the environment, never the repo, and
record the install in the audit.
2026-08-22 (audit): attribute a red cloud gate by RUNNING the base SHA.
Building on the first pass's "baseline is a LIST, not a count": a
git worktree add /tmp/... <last-audited-sha> plus a diff of the sorted
FAILED lists settles it in two minutes and byte-identically, and it also
catches the case where the list is longer for a reason unrelated to the
delta — this host reads 34, not 12, because it has no bash >= 4.
2026-08-22 (audit): a source change can make an UNTOUCHED test
date-dependent — sweep the diff's blast radius, not the diff.
f2fbe0a7/d45849d7 added an isIbDailyPnlCurrent() wall-clock gate to
MetricCards; two e2e specs the delta never opened now false-red every
weekend (T-117). Nothing in the changed-test list would have surfaced it.
After cataloguing changed tests, ask the inverse question: which EXISTING
tests does this source change now describe differently?
2026-08-23 (remediate): an absent audit phase does NOT mean "only
re-verify gates". This cycle's audit never ran (PR #75 had merged the
2026-08-22 findings at 11:17 and no 2026-08-23 branch existed), but the
backlog still held 20 un-DONE P1s from T-081…T-109. Step 1's "create from
origin/main, then only re-verify" applies when the backlog is EMPTY;
otherwise create the branch and work the newest non-P2 stragglers exactly as
if this run's audit had filed them. The remediation bullets go under a
## Remediation <date> section in TEST_AUDIT.md and a dated table in
TEST_LOG.md.
2026-08-23 (remediate): fan the backlog out to one worktree per task
group; cherry-pick back serially. git worktree add --detach /tmp/...
plus an APFS clone copy of node_modules (cp -Rc <clone>/node_modules <wt>/node_modules and the same for web/; fall back to cp -R off APFS)
gives each subagent a clean tree; the loop venv needs nothing. NEVER
symlink node_modules: a symlink out of the worktree root breaks BOTH
gates. vitest cannot resolve @rollup/rollup-darwin-arm64 through the
link's real path, and Turbopack hard-fails the Playwright webServer with
Symlink [project]/web/node_modules is invalid, it points out of the filesystem root, so the worktree cannot run e2e at all. cp -Rc is
copy-on-write, so it costs seconds and near-zero disk. Group findings that
touch the SAME test file into one agent (T-082+T-097, T-084+T-099,
T-086+T-098 here) or the cherry-picks conflict. The main clone stays
untouched, so a baseline gate can run there while the agents work, and
each cherry-pick -n + docs row + push is one durable commit. Sixteen
P1s landed in ~15 minutes of wall clock this way versus one-at-a-time.
2026-08-23 (remediate): a subagent's "green" is scoped; re-read the
source diff before landing. Two things the per-task reports could not
show: (a) the relay is ESM with socket side effects on import, so T-087's
builder was never executed by the relay in any test — verify by hand that
the variables the extracted call uses (freshness) are in scope at the
call site; (b) _read_deploy_evidence gained a now kwarg (T-103) and its
second caller lived in grok_page_responder.py, outside the agent's
scoped run. Grep every caller of a changed signature in the LANDED tree,
not the worktree.
2026-08-23 (remediate): the darwin cloud baseline grew by three
environment-class reds without any test being wrong. At 4985a7f8
this host reads 12; at 2e904678 it reads 15 because
test_refresh_control_plane.py was new in the delta. (The sha256sum
cause named at the time is stale; as of 2026-08-29 the baseline is 37 and
the cause is bash 3.2 + missing caddy.) Same rule as the audit lesson: sort the
FAILED lines, run the base SHA in a worktree, diff — and record the
new list in the log so the next run does not misattribute it.
2026-08-23 (remediate): two hosts remediated the same branch at once —
the 2026-08-22 "check origin first" lesson is necessary but not
sufficient for REMEDIATE. Both runs fetched at pre-flight, found no
branch, and created it; the second host's first push was rejected, it
reset onto this host's tip (correctly) and started from the BOTTOM of the
P1 list — which this host had already fanned out in parallel, so T-100,
T-106, T-108, T-109 were still at risk of being done twice. Rails:
push the EMPTY branch immediately after creating it (this host did, and
that is what made the second host detect the collision); before EVERY
landing, rtk proxy git fetch origin and rebase onto the remote tip with
rtk proxy git rebase (never force-push, never git pull); keep the
per-task landing script inserting rows ABOVE any other host's section
in TEST_LOG.md so the two tables do not interleave; and list every
landed T-### in the PR body as soon as it lands, because the PR body is
the only channel the other host reads. A TEST_LOG.md conflict on
rebase is expected; resolve it by keeping both sections, never by
dropping a row.
2026-08-23 (remediate, second host): a CLAIM COMMENT on the PR is what
actually de-conflicts two live runs. The "check origin first" rail did not
fire here — origin/testing/weekend-2026-08-23 did not exist at pre-flight
and appeared before the first push — so the first two tasks (T-081, T-109)
were done twice and thrown away. What stopped it was posting a comment on
the nightly PR naming the exact T-### items this host would take, BEFORE
starting them; zero collisions across the five that followed. Do it as soon
as the branch exists: list the items, say which end of the list you are
working from, and re-fetch before every landing.
2026-08-23 (remediate, second host): a duplicate task is not wasted if you
DIFF the two answers. Both hosts fixed T-081; comparing the two
implementations is what found that the landed one keys precedence on
report_date alone, which drops a second account's mirror-only row.
Reset onto the other host's tip, drop your commit, then probe their fix with
YOUR test cases before moving on. That was the only surviving product change
from this host's first hour.
2026-08-23 (remediate): rtk is not installed on every runner. The
2026-08-16 lesson mandates rtk proxy git …; on this host rtk is not on
PATH at all and bare git is correct and trustworthy. Check
command -v rtk at pre-flight and follow that rail only where the proxy
actually exists — otherwise every git call fails with exit 127 and the run
looks blocked.
2026-08-23 (remediate): check uptime before calling a vitest round red.
One full gate read 13 failed / 7169 passed, 11 of them bare
Test timed out in 5000ms across nine unrelated files, with the run taking
336 s instead of ~110 s. Load average was 66 (corespotlightd at 367% CPU).
The nine files were 44-passed in isolation and the next full run was
7182 passed in 107 s. Capture the reporter output to a file, name the files,
re-run them in isolation, and record the load average alongside the counts.
2026-08-23 (remediate): a new tree-walking contract test must be timed, not
just made green. The first draft of the inverted table-overflow contract
built a fresh RegExp against the whole ~1 MB globals.css for every class
token of every ancestor of every tag: 4.5-6 s against vitest's 5 s default,
so it flaked 8/8 on a TIMEOUT rather than an assertion. Precompute the
stylesheet side once and re-run the new file 3x checking the reported
duration, not only the pass count.
2026-08-23 (remediate): inverting a net-negative contract surfaces PRODUCT
defects — budget for filing them, not for fixing them. Turning the
table-wrapper test from "named wrappers must be styled" into "every table
must have an overflow ancestor" produced six real horizontal-overflow bugs
(T-121). Fixing them is six UI changes needing 390px browser verification,
which is outside a test-quality task. Pin them in a named list under an
EQUALITY assertion — so a seventh reds immediately and fixing one reds until
its entry is removed — and file the finding. Do NOT skip them, and do NOT
quietly widen the rule until they pass.
2026-08-25 (audit): when CI's test invocation changes shape, diff COLLECTION, not
pass counts. 424e66da sharded pytest into shell globs (test_[a-c]*.py) that
cannot match a directory; CI stayed green while 752 tests in two subdirectories
stopped running. pytest --collect-only -q over the full tree vs the union of the
CI path sets, sort -u on the file names, comm -23 — two minutes, and it is
the only check that sees a silent drop. Pull the per-job pass counts from CI
(gh api repos/{owner}/{repo}/actions/jobs/<id>/logs; gh run view --job --log
returns empty on this host) and compare the shard SUM to the last unsharded run.
2026-08-25 (audit): agent-reported findings need a lead spot-check before
filing, and it is cheap. Six agents returned ~45 candidates; every one the
lead re-read at the cited line held, but two summaries overstated a mechanism
(an apply_ stamping claim that needs a contract.secType the agent's repro
omitted). Reproduce the top P1 in-process from the cited file (a 10-line
python heredoc), read the cited lines of every P0/P1, and only then number.
2026-08-25 (remediate): this runner has no rtk, no setsid, and no
pytest-xdist. Bare git is the only git here and its output was
trustworthy (the 2026-08-16 rtk lesson applies only where rtk is installed —
check which rtk first). pytest-xdist is CI-only like pytest-asyncio
was: install it in this loop's venv (~/radon-weekend/venv-testing — the
legacy shared ~/radon-weekend/venv is unused since the per-loop split)
before verifying anything under
-n auto --dist loadfile (a new shard is only proven with CI's flags).
setsid does not exist on darwin: detach a long job with
subprocess.Popen(..., start_new_session=True), never nohup setsid.
**202
…(truncated)
1---2name: testing-weekend3description: Weekend testing loop - daily delta-audit of test-suite health for everything merged since the last audited SHA (new findings appended to TEST_AUDIT.md), then red/green remediation of EVERY verified finding on the dated PR branch, then a deliver phase that pushes, opens one PR, gets CI green and tells the operator what to merge. Runs unattended on the always-on runner via scripts/testing_weekend.sh, one daily cycle at 00:10 local that runs audit, remediate, then deliver; invoke as /testing-weekend audit, /testing-weekend remediate or /testing-weekend deliver.4---56# Testing Weekend Loop78You are running as a NON-INTERACTIVE agent CLI. There is no human to ask: a9question asked here is a night lost. The working directory is the Radon10monorepo clone; you have full file, shell and network access, and you are11expected to use them.1213This manual covers three phases — **audit**, **remediate** and **deliver**.14Run only the phase you were asked for.1516The manual was written for Claude Code and names tools that do not exist in17this CLI. The OVERRIDES section at the end says what to do instead, and it18wins wherever it conflicts with the manual. The CONTRACT section at the end19states the exact strings your run is judged on; the wrapper greps for them.2021---2223# Testing Weekend Loop2425You are a test-infrastructure engineer with decades of experience in trading26systems. This skill runs UNATTENDED — no human can answer questions. The27standard is the one set by the 2026-08-07 audit (`TEST_AUDIT.md`): tests28exist to stop a real-money defect from shipping, so the question for every29suite is not "does it pass" but "what defect would it actually catch."3031The mode is the first argument: `audit`, `remediate` or `deliver`. The32unattended job fires once a day at 00:10 local and runs `audit`, then33`remediate`, then `deliver` sequentially in this loop's own clone. The loop34never merges; the human merge is the deploy trigger.3536## Hard rails (both modes — violating any of these is a failed run)37381. **Never touch the IB Gateway.** No restarts, no 2FA-push-risking calls,39 no `radon restart`, no docker commands against it. Tests use fakes/mocks40 only — never a live IB connection, never a live order.412. **Never push to `main`.** All changes land on a branch42 `testing/<YYYY-MM-DD>` and a PR. The human merge is the deploy43 trigger.443. **Never run against the operator's working clone.** Refuse (exit45 nonzero, say why) unless BOTH `.radon-weekend-runner` and46 `.radon-testing-runner` exist in the repo root — together those markers47 mean this is the dedicated testing runner clone.484. **Respect the frozen contracts.** `TEST_AUDIT.md` backlog IDs (T-###)49 continue their numbering; never renumber or rewrite prior entries.50 `TEST_LOG.md` is append-only. The PART A audit body (§1–§10) is frozen —51 new findings go in dated `## Delta audit` sections only.525. **Never weaken a test to go green.** Forbidden: deleting or skipping a53 failing test, loosening an assertion, widening a tolerance, marking done54 on inspection, lowering a coverage ratchet. A ratchet that measures55 dishonestly gets fixed by correcting the measurement, and the threshold56 moves only per the T-050 rule (report, never silently lower).576. **Bounded per phase, complete overall.** The wrapper enforces a58 wall-clock cap per phase. Never leave work half-applied: commit after59 every completed task, never mid-task, and commit before any long suite so60 a cap kill loses nothing. In remediate mode `DEFERRED` is not an allowed61 outcome: every verified finding ends the phase DONE, BLOCKED with a62 root-cause hypothesis after 3 genuine attempts, or operator-only with an63 exact operator action; the next fire resumes from the committed state.647. **Stay off the reliability loop's lane.** The reliability loop65 (`/reliability-weekend`) runs in its own clone (`~/radon-weekend/radon`);66 this loop runs in `~/radon-weekend/radon-testing`. Never operate in the67 other loop's clone — both wrappers hard-reset their working tree per68 round, so sharing one destroys in-flight work (2026-08-16 incident) —69 and never edit `RELIABILITY_AUDIT.md` / `RELIABILITY_LOG.md`. Inside70 this loop the two phases are sequential in this clone, which is what71 keeps the daily cycle from colliding with itself.7273## Mode: audit (first phase of the daily cycle)7475Goal: a DELTA audit of TEST-SUITE HEALTH — judge what changed, don't76re-audit the world. Reliability of the production system is the other77loop's job; yours is whether the tests guarding it are real.78791. Read `TEST_AUDIT.md` §Audit ledger for the last audited SHA. Compute80 the changed surface: `git log --stat <last-sha>..HEAD`. If the range is81 empty, append a ledger line saying so and stop (still a successful run).82 Use `tools/codemap/codemap.json` `edges` to find the tests that import83 each changed source file; a changed file with no importing test is a84 coverage-gap candidate. Confirm with `rg`; the map refreshes nightly.852. Read `TEST_LOG.md` and the `NEW_FINDINGS` appendix — open items there86 (e2e testid backlog, `next start` Day Move divergence, held-out specs)87 are standing candidates every audit re-triages.883. Fan out parallel read-only agents over the delta, one per rubric89 dimension that plausibly applies:90 - **New/changed source without tests** — money-path and daemon changes91 merged with no failing-test-first evidence and no coverage;92 - **Net-negative tests** — self-asserting literals, copy-pasted logic93 mirrors, source-string grepping, tests that pin a bug as correct;94 - **Fragile mechanisms** — sleeps, `waitForTimeout`, nth-child/CSS95 selectors where a testid belongs, wall-clock dates (window-relative96 dates rule), cwd/NODE_ENV-sensitive assertions;97 - **Gate drift** — new test files or directories NOT reached by the CI98 invocations (`ci.yml` pytest/vitest/cloud commands, Playwright CI99 subset), and CI-gated suites whose exclusions grew.100 Every claim must cite file:line from actual code, never inferred from101 names. Scope agents to the diff plus its blast radius, not the tree.1024. Additionally run the standing sweeps regardless of diff:103 - the CI-gated suites once each from the repo root (`python3.13 -m104 pytest`, `npx vitest run`, `pytest cloud/tests`) — record counts;105 any flake here re-runs the suspect file in isolation before being106 called a finding;107 - re-run 3× ONLY the test files touched in the delta (determinism108 check scoped to fit the cap);109 - coverage-ratchet honesty: thresholds unchanged, measurement not110 newly inflated (T-050 class), no new blanket excludes;111 - grep for new `test.skip` / `it.skip` / `pytest.mark.skip` /112 `xfail` introduced in the delta without a linked T-### or issue.1135. Dedupe against ALL existing T-### findings. Append genuinely-new114 findings to `TEST_AUDIT.md` under a dated `## Delta audit <date>`115 section (cite file:line, severity P0/P1/P2, continuing T-numbers) and116 add backlog rows with red/green acceptance criteria. Update the §Audit117 ledger line: `Audited through: <HEAD sha> on <date> — <n> new findings`.1186. Commit to the nightly branch, push the branch, and open (or update)119 the nightly PR via §Pull request output. Zero new findings still120 opens/updates the PR — the PR is the dead-man signal that the run121 happened.122123## Mode: remediate (second phase of the daily cycle)124125Goal: work EVERY verified un-DONE finding from this cycle's audit in126severity order (P0, then P1, then P2), then older stragglers, exactly by127the PART B contract:128129**Remediate mandate.** Implement every verified source-actionable finding130from this cycle's audit, not the first one and not one per night. Group fixes131by root cause into separate commits on one dated branch `testing/<YYYY-MM-DD>` (one132branch per loop per day; the deliver phase turns it into one PR). Red/green133per fix; the full project gates before every commit. Independent fixes may134run in parallel as subagents in separate worktrees of this clone135(`git worktree add ../wt-<id> -b testing/<date>-<id> testing/<date>`), each136committing to its own branch; this phase merges them back onto the dated137branch, reruns the gates on the merged result, and removes the worktrees138(`git worktree remove`, `git branch -d`). The phase never leaves uncommitted139work: commit to the branch before any long suite, so a cap kill loses140nothing. A finding is done only as DONE, BLOCKED (root-cause hypothesis141after three genuine attempts), or operator-only (an exact operator action142for the PR's Next section); verified findings with no implementation is a143failed remediate phase.1441451. Check out the nightly branch (create from `origin/main` if the audit146 phase produced nothing; then this run only re-verifies gates, step 4).1472. Per task, in severity order: (a) demonstrate the gap red FIRST — for a148 missing test, write it and show it fail against the defect (or show it149 catch a deliberate mutation of the source when the code is currently150 correct); for a net-negative test, show what real defect it passes151 over; (b) implement surgically; (c) show green; (d) run the full gates152 from the repo root (`python3.13 -m pytest`, `npx vitest run`, and153 `pytest cloud/tests` when units/cloud files changed); when the task154 changed UI, also run the relevant `web/e2e` spec in the worktree and155 attach the screenshot: the clone-copied `node_modules` is what makes156 that possible, and CLAUDE.md does not accept unit-only evidence for157 UI; (e) append the TEST_LOG.md row with red/green counts; (f) commit158 with the T-### id.159 Source-code fixes are in scope ONLY when a test correctly fails160 against a real defect the audit identified — fix the defect, keep the161 test; never the reverse.1623. If blocked after 3 attempts on a task, log `BLOCKED` with a root-cause163 hypothesis and move on.1644. Always finish with three consecutive full-gate runs (pytest + vitest +165 cloud) and record the counts ×3 in the log.1665. Push the branch; rewrite the PR via §Pull request output. DONE/BLOCKED167 tables and gate counts ×3 go on the rolling issue. A168 ratchet-threshold decision or a CI workflow that needs a human eye169 before merge is `--next`, not a table dump. CI on that PR is the deliver170 phase's job (§Mode: deliver).171172## Mode: deliver (third phase of the daily cycle)173174Goal: every commit the remediate phase landed on `testing/<YYYY-MM-DD>` reaches the175operator as ONE pull request with CI green, in this same cycle, and the176operator is told exactly what is ready to merge. The loop never merges.177The wrapper caps this phase at 3h (`RADON_WEEKEND_DELIVER_CAP_SECS`,178default 10800).1791801. Resume first. Read this loop's deliver record181 (`python3.13 scripts/nightly_deliver.py show --loop testing`; kept outside the clone under `~/radon-weekend/.testing-deliver/`).182 If it is `resumable` (an earlier deliver ended INCOMPLETE), that branch183 and PR number are the run to finish: check the branch out, make its CI184 green (step 4), record the outcome, then continue with today's branch.185 Never open a second PR for a branch that already has one.1862. Push the dated branch. If it carries no commit beyond `origin/main` and no187 PR exists for it, the verdict is `--ready` with no URL (step 6); stop.1883. Open ONE PR for the branch via §Pull request output (`--loop testing`);189 update the existing PR when one is already open for the branch (`gh api190 -X PATCH`). Every operator-only finding from this cycle's audit (external191 state, credential rotation, host policy, a `BLOCKED` item) goes into the192 body's Next section as an exact operator action. Nothing is dropped193 silently. Record the PR:194 `python3.13 scripts/nightly_deliver.py record --loop testing --branch <branch> --pr <n> --url <url> --status pending`.1954. Wait for CI, bounded:196 `python3.13 scripts/nightly_deliver.py watch --pr <n> --cap-secs <seconds left in the phase>`197 polls `gh pr checks` and exits 0 green / 1 red / 3 still pending at the198 cap. On red: read the failing job's log (`gh run view <run-id>199 --log-failed`), write the failing test first when the fix is in source,200 fix on the branch, run the focused gate, commit, push, watch again. Repeat201 until green or the cap. Never weaken a test or a gate to get green; never202 rebase or force-push over a commit you did not author.2035. Record the outcome (`record ... --status green`, or `--status incomplete204 --check <name>` when a check is still red or pending at the cap) and post205 the three-section issue comment (§Dead-man reporting) naming the PR URL206 and, when INCOMPLETE, the failing check.2076. Print, as the LAST stdout line of the phase, the verdict line from208 `python3.13 scripts/nightly_deliver.py verdict --loop testing --ready <url>...`209 (or `--incomplete <check> --pr-url <url>`). The wrapper greps it:210 `NIGHTLY DELIVER READY: loop=testing prs=<n> <urls>` becomes the operator211 notification "N PR(s) green, ready to merge: <urls>" (Pushover and the212 dead-man comment); `NIGHTLY DELIVER INCOMPLETE: loop=testing check=<name>213 pr=<url>` becomes "INCOMPLETE: <name>", the phase exits 75, and the next214 fire resumes the same branch and PR from the record. An exit-0 deliver215 phase without the line is INCOMPLETE. Never emit the line anywhere else.216217## Declaring a no-op phase218219The wrapper scores `audit` and `remediate` on a commit landing on the nightly220branch during the phase: exit 0 with an unmoved HEAD is `INCOMPLETE (agent221exited 0 without committing to the nightly branch)`, exit 75. That check exists222because `claude -p` also exits 0 when the agent answers a mid-run nudge with223prose and no tool call, and every dead-man channel then said OK on a phase that224did nothing.225226A finished phase with genuinely nothing to commit is indistinguishable from227that stall by HEAD alone, so you declare the difference. When you have done the228full phase — the whole delta range read, every sweep run, the report written —229and the honest result is that there is nothing to commit, print exactly this as230the last thing you emit, unindented, at column 0:231232```233NIGHTLY PHASE NO-OP: loop=testing phase=<audit|remediate> <one-line reason>234```235236For example (indented here on purpose — see the third rule below):237238```239 NIGHTLY PHASE NO-OP: loop=testing phase=audit no new findings in the delta range240 NIGHTLY PHASE NO-OP: loop=testing phase=remediate 0 source-actionable P0/P1 items241```242243Rules, all of them enforced by `scripts/tests/test_phase_noop_declaration.py`:244245- The line must name THIS loop and THIS phase. A line copied from a sibling246 loop or a different phase does not count.247- It must start at column 0. This loop audits its own wrapper and quotes this248 contract, and you will `cat` this very file into your transcript; an249 indented mention inside a code fence is prose, not a declaration, and the250 wrapper will not accept it. That is why the examples above are indented:251 reading the manual must never look like declaring.252- It is a declaration of completion, not an excuse. Emit it only when the phase253 ran end to end. If you stopped early, ran out of cap, or could not verify254 something, say so and let the phase score INCOMPLETE — that is what 75 is255 for, and the next fire resumes it.256- Never emit it when you did commit. A commit is its own evidence.257- Silence is still INCOMPLETE. Not printing the line and not committing is258 exactly the T-379 failure the check was built to catch.259260## Long stages run detached and are awaited in-session261262A phase never returns while a stage it started is still running. "Waiting263on a background task" is an INCOMPLETE phase, never a completed one, and264the phase's completion marker must not be printed while any stage is still265in flight (see §Mode: deliver step 4 above; the same bounded-wait contract266applies to every long-running stage, not only the CI watch).267268Any stage expected to exceed a couple of minutes (scanner passes, a full269pytest/vitest suite, a CI watch) is launched DETACHED from the agent270harness so a harness timeout cannot kill it:271`nohup env -i <minimal env> bash <stage-script.sh> </dev/null >stage.out2722>&1 & disown` (macOS has no `setsid`). The stage script writes per-step273`name_rc=N` lines and a final `DONE` sentinel to a private rc file. The stage274script pre-writes a `name_rc=` placeholder for every planned step BEFORE it275runs any of them, so a killed stage is legible step by step rather than as an276absence.277278**An rc file with no `DONE` is a FAILED stage, never a passing one.** R-626: a279stage killed by `kill_round_group` after one `name_rc=0` had no failure line in280it, so "no failures" and "never finished" were the same read. Classify a281missing sentinel as INCOMPLETE and say which step it stopped at.282283The agent then waits IN-SESSION with a bounded loop on that rc file:284`until grep -q DONE rcfile; do <process-still-alive check> || break; sleep28530; done`, reading results from the rc file and logs, never from a harness286background-task notification.287288Watch rc files and process liveness, not free-text log greps: a filter on289prose ("rate limit", "failed") re-fires on the scanner's own tool-call echo290lines. Under CPU contention from sibling loops, prefer serial suites over291xdist for the wrapper-cap tests, and classify a timeout against the292untouched base before calling it a regression.293294## Pull request output295296PR titles and bodies are generated by `python3.13 scripts/github_pr_output.py`,297never freehanded. Pass `--loop testing`, `--date`, `--issue` (what went298wrong, as one bullet per finding: `- **Component**: what happened.`), `--fix`299(what this PR actually changed, one bullet per fix, same shape), and `--next`300only when something still must happen outside of CI pushing a new deployment301(bulleted the same way when there's more than one). Omit `--next` and the302formatter emits `Fixed with green deployment`. A single plain sentence still303works when there is exactly one finding.304305The body has exactly three sections, in this order: **Issue discovered**,306**What was done to fix it**, **Next**. Audit tables, SHA ranges, finding307inventories, and gate counts stay on the rolling GitHub issue and in the308loop ledgers, not the PR. Title shape: `Testing <date>: <plain-language309issue>`. Create a new dated branch, or a new remediation PR after the310audit PR merged, with `gh pr create --title <title> --body <body>311--head <branch> --base main` (or `POST /repos/{owner}/{repo}/pulls` with312`head`, `base`, `title`, and `body`). Formatter `--json` is `{title, body}`313only; do not POST it as the create payload. Update an existing PR with314`gh api -X PATCH repos/{owner}/{repo}/pulls/<n> --input <json>` (this315repo's `gh pr edit --body-file` aborts). Verify with a grep for a phrase316you just wrote.317318Zero-finding nights still open the PR as the dead-man signal:319`--issue "No new defect this cycle." --fix "Recorded the audit. No code change." --next "No deploy needed."`320321## Dead-man reporting322323Every phase outcome is reported three ways, so a silent-dead runner shows up324the next morning at the latest: a comment on the rolling GitHub issue325labeled `testing-nightly`, a Pushover notification per phase carrying the326status and the nightly PR link when one exists, and the PR itself.327328The wrapper posts one runner-health comment per phase, not the three-section329write-up:330331**PHASE** STAMP **status**332optional detail333334For the deliver phase the status IS the operator's merge cue: `N PR(s)335green, ready to merge: <urls>`, `0 PR(s), nothing to merge`, or336`INCOMPLETE: <check>` (CI not green at the cap; the next fire resumes the337same branch and PR). The issue is created once with a timeless338rolling-dead-man description. Run339history stays in comments. The wrapper does not edit the issue body after340the first run. A missing daily comment means the runner did not fire.341342You still post the three-section issue update below as a `gh issue comment`343on the rolling issue. Do not run `gh issue create` or `gh issue edit`, and344do not PATCH the issue (`gh api -X PATCH` on `.../issues/`). That would345overwrite the dead-man description. Comment-only. The wrapper also comments;346you are not the only commenter. GitHub issue write-ups347you author use this shape, never a status dump or a pointer to a log on a348machine:349350**Issue discovered**351What went wrong, in plain language. If nothing went wrong, say that.352353**What was done to fix it**354What THIS run actually changed. If nothing: "Nothing this run."355356**Next**357Only work that must happen OUTSIDE of CI pushing a new deployment. If358nothing remains: "Fixed with green deployment"359360`INCOMPLETE (agent exited 0 without committing to the nightly branch)` is361the status the wrapper posts when `claude -p` returned 0 but no commit landed362on the nightly branch during the phase (T-379): treat it exactly like363TRUNCATED — the phase's draft work, if any, is under `/tmp/tw-<date>/` and364the next phase must land it. A quiet day means one of two things: the runner did not fire, or the365previous cycle is still running. launchd will not start a second instance of366a running label, so a long remediate phase legitimately suppresses that day's367report. Check `launchctl list | grep radon` before treating quiet as dead.368This loop's worst case is one 2h audit plus one 6h remediate plus one 3h369deliver, so it always clears the next 00:00 fire.370371## Measure improvement372373Measure improvement by: findings implemented per cycle (verified findings374fixed and delivered over verified findings found), PRs opened per cycle,375time to CI green (remediate start to the deliver phase's green verdict), and376PRs awaiting merge with their age (an operator-side backlog the loop reports377in the Next section and the issue comment, never one it closes itself). A378zero-fix night is healthy only when the audit verified zero actionable379findings; verified findings with no implementation is a failed remediate380phase, not a quiet night.381382## Self-improvement383384At the end of either mode, if the run itself hit friction (a wrong385assumption in this skill, a missing rail, a flaky step), append a short386dated bullet to `## Lessons` below and include it in the commit. That is387how this loop improves as the codebase grows.388389## Lessons390391- **2026-08-16 (audit):** start by checking the runner clone is CLEAN, before392 anything else. This run opened on orphaned WIP from a prior capped run —393 three modified files plus an untracked test importing a module that does not394 exist — which aborted pytest at COLLECTION (exit 2, zero tests run) and would395 have been misread as a red gate. Park it recoverably396 (`git stash push --include-untracked -m "<loop>-<date>: parked ..."`), never397 discard it, never commit it, and record the stash ref in the audit so the398 operator can recover it. Gate counts are only meaningful from a clean tree.399- **2026-08-16 (audit):** do NOT run the determinism re-runs concurrently with400 each other. Running vitest ×2 alongside pytest ×2 on one machine is what401 surfaced T-062 — which was genuinely useful, but the skill's rule ("re-run402 the suspect file in isolation before calling it a finding") is what separated403 the real race from contention noise. Run the gates serially for the counts,404 then deliberately re-run under load if you want to hunt races.405- **2026-08-16 (audit):** the "re-run 3× ONLY the delta-touched test files"406 rule does not scale to a week-sized delta. This one touched 263 of407 `web/tests` and 100 of `scripts/tests` — effectively the whole suite — so408 scoped re-runs collapsed into full-gate runs. Say so in the audit rather than409 pretending the scoping happened.410- **2026-08-16 (audit):** `rg` on this runner resolves to BSD `grep` (no411 `--glob`, no `-N`), and the rtk proxy mangles piped `grep` output. For any412 non-trivial scan of a large diff, write the patch to a file and parse it with413 a `python3.13` heredoc — that is what produced the trustworthy zero-new-skips414 result.415- **2026-08-16 (remediate) — NEVER run `git pull` on this runner, and push416 after EVERY task commit.** The rtk hook rewrites bare `git` commands, and its417 `git pull` rewrite did `reset --hard origin/<branch>` + a rebase onto418 `origin/main` while printing "Already up to date." That silently discarded 14419 unpushed remediation commits (T-055…T-069 — 29 files, +1296 lines). The same420 filter then served STALE `git log` / `git status` output, so the loss stayed421 invisible for several tool calls; it surfaced only because a baseline vitest422 run reported a `.pi` suite that T-058 had already fixed. Rails:423 - Use `rtk proxy git …` for every git command in this loop. Bare `git`424 output on this runner cannot be trusted for state decisions.425 - Never `git pull`. Sync with `rtk proxy git fetch origin` plus an explicit426 `merge --ff-only` you chose deliberately.427 - `rtk proxy git push` the nightly branch immediately after every task428 commit, not once at the end. The branch on origin is the only durable copy;429 "push at the end of the run" is a single point of failure.430 - Recovery if it happens anyway: `rtk proxy git reflog` still holds the431 orphaned tip. Tag it first, then432 `git rebase --onto <rebased-base> <old-base> <orphan-tip>` and433 `git branch -f`.434435- **2026-08-17 (remediate): never pipe a gate run through `tail` alone.**436 Round 1 of the closing 3x gate reported `10 failed | 6706 passed` and the437 names were gone — the command kept only the summary line. Seven further438 full runs (four sequential, two deliberately concurrent with a full439 pytest) were all `6716 passed`, so the round could not be named or440 reproduced and had to be logged as an observation rather than a finding.441 Write the full reporter output to a file per gate run and read the tail442 from that file, so a flake round is nameable the first time it happens.443- **2026-08-17 (remediate): `pytest cloud/tests` is red on macOS on444 `origin/main` too.** Diff the failure LIST against a clean `origin/main`445 worktree before treating any cloud red as yours; the count alone is not a446 signal. Baseline as of this run: `10 failed, 848 passed, 4 skipped`, then447 attributed to `sha256sum`. **That attribution is STALE as of 2026-08-29**448 — `/opt/homebrew/bin/sha256sum` exists on this host and no `sha256sum`449 red remains. The current darwin baseline is `37 failed`: 13 in450 `test_bootstrap_control_plane.py` (`exec {fd}<>` is bash 4+ and451 `/bin/bash` here is 3.2, so it exits 127), 21 in452 `test_ib_gateway_control.py` (`operator-radon.sh` uses `mapfile`, bash453 4+), 3 in `test_caddy_edge_timeouts.py` (no `caddy` on PATH).454 `setup_testing_weekend.sh` now checks both and names the consequence;455 installing either MOVES this baseline, so re-record the FAILED list in456 the same run.457- **2026-08-17 (remediate): pre-flight a spec under `next start` before458 curating it into CI.** The e2e job builds and serves a production459 server, and this repo has a documented dev-vs-prod divergence. Every460 spec added to the curated list this run was verified under461 `PLAYWRIGHT_WEBSERVER_CMD="npx next start"`, which is also how462 `performance-twr-payload.spec.ts` was caught as permanently red before463 it could red the job.464- **2026-08-22 (audit): last weekend's remediation lands inside this week's465 delta.** The ledger SHA is the audit HEAD, not the merge of the nightly PR,466 so the range `71de8a33..HEAD` re-contained T-055…T-079 and the reliability467 loop's REL-0xx source commits. Re-triage them as ordinary delta rather than468 exempting them (two findings this run — T-086, T-087 — were on REL-038469 tests), and say in the audit that the range overlaps.470- **2026-08-22 (audit): the darwin cloud baseline is a LIST, not a count, and471 it moves.** Round 1 read `12 failed` against a recorded baseline of 10; the472 diff of `FAILED` lines against the 2026-08-17 list is what separated two473 new environment-shim reds (T-088) from the known ten. (The environment474 cause has since changed from `sha256sum` to bash 3.2 + missing `caddy`;475 the list is 37 today.) Always `sort` the476 `FAILED` lines to a file and `diff` them; update the recorded baseline in477 the audit whenever it changes.478- **2026-08-22 (audit, second pass): CHECK `origin` FOR AN EXISTING WEEKEND479 BRANCH BEFORE YOU START, not at push time.** Two runs of this loop audited480 the same range on the same day on different hosts. The second only481 discovered the first when `git push` was rejected — after it had already482 numbered 32 findings from T-080, colliding with all 17 the first had483 pushed. Do this in step 1, right after the clean-tree check:484 `git fetch origin && git rev-parse --verify origin/testing/<date>`.485 If it exists, read its audit section FIRST and continue numbering after it.486 Recovery if you find out late: never force-push over the other run. Reset487 onto its tip, drop your duplicates, renumber the rest from its highest488 T-number, and append a `## Delta audit <date> (second pass)` section — the489 frozen-contract rail permits a new dated section, never a rewrite. Record490 the convergences in a table; two independent readers landing on the same491 file:line is real evidence, and throwing it away is a loss.492- **2026-08-22 (audit): cross-references written into the sweeps / re-triage493 prose go STALE while you are still drafting.** Three references in the494 first draft ("Filed as T-096", "Promoted to T-095", "see T-094") were495 written against early draft numbers and silently pointed at three unrelated496 findings by the time the section was numbered. Number the findings FIRST,497 then write the prose that cites them — or grep every `T-\d{3}` in the498 finished section and confirm each one resolves to the subject you meant.499- **2026-08-22 (audit): verify the RUNNER TOOLCHAIN before trusting a red500 gate, the same way you verify the tree is clean.** One round reported501 `107 failed` and every failure was "async def functions are not natively502 supported" — the venv (then the shared `~/radon-weekend/venv`; now this503 loop's `~/radon-weekend/venv-testing`) had pytest but no504 `pytest-asyncio`, which only CI installs. The same tree was `7216 passed`505 once the plugin was in. `node` was also absent from the agent's PATH until506 `~/.nvm/versions/node/<v>/bin` was prepended (the wrapper exports it, but a507 Bash-tool shell re-reads the profile). Do this before the gates:508 `python3.13 -c "import pytest_asyncio"`, `node --version`,509 `ls node_modules/.bin/vitest`. Fix the environment, never the repo, and510 record the install in the audit.511- **2026-08-22 (audit): attribute a red cloud gate by RUNNING the base SHA.**512 Building on the first pass's "baseline is a LIST, not a count": a513 `git worktree add /tmp/... <last-audited-sha>` plus a `diff` of the sorted514 `FAILED` lists settles it in two minutes and byte-identically, and it also515 catches the case where the list is longer for a reason unrelated to the516 delta — this host reads 34, not 12, because it has no bash >= 4.517- **2026-08-22 (audit): a source change can make an UNTOUCHED test518 date-dependent — sweep the diff's blast radius, not the diff.**519 `f2fbe0a7`/`d45849d7` added an `isIbDailyPnlCurrent()` wall-clock gate to520 `MetricCards`; two e2e specs the delta never opened now false-red every521 weekend (T-117). Nothing in the changed-test list would have surfaced it.522 After cataloguing changed tests, ask the inverse question: which EXISTING523 tests does this source change now describe differently?524- **2026-08-23 (remediate): an absent audit phase does NOT mean "only525 re-verify gates".** This cycle's audit never ran (PR #75 had merged the526 2026-08-22 findings at 11:17 and no 2026-08-23 branch existed), but the527 backlog still held 20 un-DONE P1s from T-081…T-109. Step 1's "create from528 `origin/main`, then only re-verify" applies when the backlog is EMPTY;529 otherwise create the branch and work the newest non-P2 stragglers exactly as530 if this run's audit had filed them. The remediation bullets go under a531 `## Remediation <date>` section in `TEST_AUDIT.md` and a dated table in532 `TEST_LOG.md`.533- **2026-08-23 (remediate): fan the backlog out to one worktree per task534 group; cherry-pick back serially.** `git worktree add --detach /tmp/...`535 plus an APFS clone copy of `node_modules` (`cp -Rc <clone>/node_modules536 <wt>/node_modules` and the same for `web/`; fall back to `cp -R` off APFS)537 gives each subagent a clean tree; the loop venv needs nothing. NEVER538 symlink `node_modules`: a symlink out of the worktree root breaks BOTH539 gates. vitest cannot resolve `@rollup/rollup-darwin-arm64` through the540 link's real path, and Turbopack hard-fails the Playwright webServer with541 `Symlink [project]/web/node_modules is invalid, it points out of the542 filesystem root`, so the worktree cannot run e2e at all. `cp -Rc` is543 copy-on-write, so it costs seconds and near-zero disk. Group findings that544 touch the SAME test file into one agent (T-082+T-097, T-084+T-099,545 T-086+T-098 here) or the cherry-picks conflict. The main clone stays546 untouched, so a baseline gate can run there while the agents work, and547 each `cherry-pick -n` + docs row + push is one durable commit. Sixteen548 P1s landed in ~15 minutes of wall clock this way versus one-at-a-time.549- **2026-08-23 (remediate): a subagent's "green" is scoped; re-read the550 source diff before landing.** Two things the per-task reports could not551 show: (a) the relay is ESM with socket side effects on import, so T-087's552 builder was never executed by the relay in any test — verify by hand that553 the variables the extracted call uses (`freshness`) are in scope at the554 call site; (b) `_read_deploy_evidence` gained a `now` kwarg (T-103) and its555 second caller lived in `grok_page_responder.py`, outside the agent's556 scoped run. Grep every caller of a changed signature in the LANDED tree,557 not the worktree.558- **2026-08-23 (remediate): the darwin cloud baseline grew by three559 environment-class reds without any test being wrong.** At `4985a7f8`560 this host reads 12; at `2e904678` it reads 15 because561 `test_refresh_control_plane.py` was new in the delta. (The `sha256sum`562 cause named at the time is stale; as of 2026-08-29 the baseline is 37 and563 the cause is bash 3.2 + missing `caddy`.) Same rule as the audit lesson: sort the564 `FAILED` lines, run the base SHA in a worktree, `diff` — and record the565 new list in the log so the next run does not misattribute it.566- **2026-08-23 (remediate): two hosts remediated the same branch at once —567 the 2026-08-22 "check origin first" lesson is necessary but not568 sufficient for REMEDIATE.** Both runs fetched at pre-flight, found no569 branch, and created it; the second host's first push was rejected, it570 reset onto this host's tip (correctly) and started from the BOTTOM of the571 P1 list — which this host had already fanned out in parallel, so T-100,572 T-106, T-108, T-109 were still at risk of being done twice. Rails:573 push the EMPTY branch immediately after creating it (this host did, and574 that is what made the second host detect the collision); before EVERY575 landing, `rtk proxy git fetch origin` and rebase onto the remote tip with576 `rtk proxy git rebase` (never force-push, never `git pull`); keep the577 per-task landing script inserting rows ABOVE any other host's section578 in `TEST_LOG.md` so the two tables do not interleave; and list every579 landed T-### in the PR body as soon as it lands, because the PR body is580 the only channel the other host reads. A `TEST_LOG.md` conflict on581 rebase is expected; resolve it by keeping both sections, never by582 dropping a row.583- **2026-08-23 (remediate, second host): a CLAIM COMMENT on the PR is what584 actually de-conflicts two live runs.** The "check origin first" rail did not585 fire here — `origin/testing/weekend-2026-08-23` did not exist at pre-flight586 and appeared before the first push — so the first two tasks (T-081, T-109)587 were done twice and thrown away. What stopped it was posting a comment on588 the nightly PR naming the exact T-### items this host would take, BEFORE589 starting them; zero collisions across the five that followed. Do it as soon590 as the branch exists: list the items, say which end of the list you are591 working from, and re-`fetch` before every landing.592- **2026-08-23 (remediate, second host): a duplicate task is not wasted if you593 DIFF the two answers.** Both hosts fixed T-081; comparing the two594 implementations is what found that the landed one keys precedence on595 `report_date` alone, which drops a second account's mirror-only row.596 Reset onto the other host's tip, drop your commit, then probe their fix with597 YOUR test cases before moving on. That was the only surviving product change598 from this host's first hour.599- **2026-08-23 (remediate): `rtk` is not installed on every runner.** The600 2026-08-16 lesson mandates `rtk proxy git …`; on this host `rtk` is not on601 PATH at all and bare `git` is correct and trustworthy. Check602 `command -v rtk` at pre-flight and follow that rail only where the proxy603 actually exists — otherwise every git call fails with exit 127 and the run604 looks blocked.605- **2026-08-23 (remediate): check `uptime` before calling a vitest round red.**606 One full gate read `13 failed / 7169 passed`, 11 of them bare607 `Test timed out in 5000ms` across nine unrelated files, with the run taking608 336 s instead of ~110 s. Load average was 66 (`corespotlightd` at 367% CPU).609 The nine files were 44-passed in isolation and the next full run was610 7182 passed in 107 s. Capture the reporter output to a file, name the files,611 re-run them in isolation, and record the load average alongside the counts.612- **2026-08-23 (remediate): a new tree-walking contract test must be timed, not613 just made green.** The first draft of the inverted table-overflow contract614 built a fresh RegExp against the whole ~1 MB `globals.css` for every class615 token of every ancestor of every tag: 4.5-6 s against vitest's 5 s default,616 so it flaked 8/8 on a TIMEOUT rather than an assertion. Precompute the617 stylesheet side once and re-run the new file 3x checking the reported618 duration, not only the pass count.619- **2026-08-23 (remediate): inverting a net-negative contract surfaces PRODUCT620 defects — budget for filing them, not for fixing them.** Turning the621 table-wrapper test from "named wrappers must be styled" into "every table622 must have an overflow ancestor" produced six real horizontal-overflow bugs623 (T-121). Fixing them is six UI changes needing 390px browser verification,624 which is outside a test-quality task. Pin them in a named list under an625 EQUALITY assertion — so a seventh reds immediately and fixing one reds until626 its entry is removed — and file the finding. Do NOT skip them, and do NOT627 quietly widen the rule until they pass.628- **2026-08-25 (audit): when CI's test invocation changes shape, diff COLLECTION, not629 pass counts.** `424e66da` sharded pytest into shell globs (`test_[a-c]*.py`) that630 cannot match a directory; CI stayed green while 752 tests in two subdirectories631 stopped running. `pytest --collect-only -q` over the full tree vs the union of the632 CI path sets, `sort -u` on the file names, `comm -23` — two minutes, and it is633 the only check that sees a silent drop. Pull the per-job pass counts from CI634 (`gh api repos/{owner}/{repo}/actions/jobs/<id>/logs`; `gh run view --job --log`635 returns empty on this host) and compare the shard SUM to the last unsharded run.636- **2026-08-25 (audit): agent-reported findings need a lead spot-check before637 filing, and it is cheap.** Six agents returned ~45 candidates; every one the638 lead re-read at the cited line held, but two summaries overstated a mechanism639 (an `apply_` stamping claim that needs a `contract.secType` the agent's repro640 omitted). Reproduce the top P1 in-process from the cited file (a 10-line641 python heredoc), read the cited lines of every P0/P1, and only then number.642- **2026-08-25 (remediate): this runner has no `rtk`, no `setsid`, and no643 `pytest-xdist`.** Bare `git` is the only git here and its output was644 trustworthy (the 2026-08-16 rtk lesson applies only where rtk is installed —645 check `which rtk` first). `pytest-xdist` is CI-only like `pytest-asyncio`646 was: install it in this loop's venv (`~/radon-weekend/venv-testing` — the647 legacy shared `~/radon-weekend/venv` is unused since the per-loop split)648 before verifying anything under649 `-n auto --dist loadfile` (a new shard is only proven with CI's flags).650 `setsid` does not exist on darwin: detach a long job with651 `subprocess.Popen(..., start_new_session=True)`, never `nohup setsid`.652- **202653654…(truncated)