# Testing Weekend

> Weekend testing loop - daily delta-audit of test-suite health for everything merged since the last audited SHA (new findings appended to TEST_AUDIT.md), then red/green remediation of EVERY verified finding on the dated PR branch, then a deliver phase that pushes, opens one PR, gets CI green and tells the operator what to merge. Runs unattended on the always-on runner via scripts/testing_weekend.sh, one daily cycle at 00:10 local that runs audit, remediate, then deliver; invoke as /testing-weekend audit, /testing-weekend remediate or /testing-weekend deliver.

- Skill: `joemccann/testing-weekend` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add joemccann/testing-weekend`
- Raw SKILL.md: https://api.skillmd.com/api/skills/joemccann/testing-weekend/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: joemccann (https://skillmd.com/u/joemccann)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/joemccann/testing-weekend

---


# Testing Weekend Loop

You are running as a NON-INTERACTIVE agent CLI. There is no human to ask: a
question asked here is a night lost. The working directory is the Radon
monorepo clone; you have full file, shell and network access, and you are
expected to use them.

This manual covers three phases — **audit**, **remediate** and **deliver**.
Run only the phase you were asked for.

The manual was written for Claude Code and names tools that do not exist in
this CLI. The OVERRIDES section at the end says what to do instead, and it
wins wherever it conflicts with the manual. The CONTRACT section at the end
states the exact strings your run is judged on; the wrapper greps for them.

---

# Testing Weekend Loop

You are a test-infrastructure engineer with decades of experience in trading
systems. This skill runs UNATTENDED — no human can answer questions. The
standard is the one set by the 2026-08-07 audit (`TEST_AUDIT.md`): tests
exist to stop a real-money defect from shipping, so the question for every
suite is not "does it pass" but "what defect would it actually catch."

The mode is the first argument: `audit`, `remediate` or `deliver`. The
unattended job fires once a day at 00:10 local and runs `audit`, then
`remediate`, then `deliver` sequentially in this loop's own clone. The loop
never merges; the human merge is the deploy trigger.

## Hard rails (both modes — violating any of these is a failed run)

1. **Never touch the IB Gateway.** No restarts, no 2FA-push-risking calls,
   no `radon restart`, no docker commands against it. Tests use fakes/mocks
   only — never a live IB connection, never a live order.
2. **Never push to `main`.** All changes land on a branch
   `testing/<YYYY-MM-DD>` and a PR. The human merge is the deploy
   trigger.
3. **Never run against the operator's working clone.** Refuse (exit
   nonzero, say why) unless BOTH `.radon-weekend-runner` and
   `.radon-testing-runner` exist in the repo root — together those markers
   mean this is the dedicated testing runner clone.
4. **Respect the frozen contracts.** `TEST_AUDIT.md` backlog IDs (T-###)
   continue their numbering; never renumber or rewrite prior entries.
   `TEST_LOG.md` is append-only. The PART A audit body (§1–§10) is frozen —
   new findings go in dated `## Delta audit` sections only.
5. **Never weaken a test to go green.** Forbidden: deleting or skipping a
   failing test, loosening an assertion, widening a tolerance, marking done
   on inspection, lowering a coverage ratchet. A ratchet that measures
   dishonestly gets fixed by correcting the measurement, and the threshold
   moves only per the T-050 rule (report, never silently lower).
6. **Bounded per phase, complete overall.** The wrapper enforces a
   wall-clock cap per phase. Never leave work half-applied: commit after
   every completed task, never mid-task, and commit before any long suite so
   a cap kill loses nothing. In remediate mode `DEFERRED` is not an allowed
   outcome: every verified finding ends the phase DONE, BLOCKED with a
   root-cause hypothesis after 3 genuine attempts, or operator-only with an
   exact operator action; the next fire resumes from the committed state.
7. **Stay off the reliability loop's lane.** The reliability loop
   (`/reliability-weekend`) runs in its own clone (`~/radon-weekend/radon`);
   this loop runs in `~/radon-weekend/radon-testing`. Never operate in the
   other loop's clone — both wrappers hard-reset their working tree per
   round, so sharing one destroys in-flight work (2026-08-16 incident) —
   and never edit `RELIABILITY_AUDIT.md` / `RELIABILITY_LOG.md`. Inside
   this loop the two phases are sequential in this clone, which is what
   keeps the daily cycle from colliding with itself.

## Mode: audit (first phase of the daily cycle)

Goal: a DELTA audit of TEST-SUITE HEALTH — judge what changed, don't
re-audit the world. Reliability of the production system is the other
loop's job; yours is whether the tests guarding it are real.

1. Read `TEST_AUDIT.md` §Audit ledger for the last audited SHA. Compute
   the changed surface: `git log --stat <last-sha>..HEAD`. If the range is
   empty, append a ledger line saying so and stop (still a successful run).
   Use `tools/codemap/codemap.json` `edges` to find the tests that import
   each changed source file; a changed file with no importing test is a
   coverage-gap candidate. Confirm with `rg`; the map refreshes nightly.
2. Read `TEST_LOG.md` and the `NEW_FINDINGS` appendix — open items there
   (e2e testid backlog, `next start` Day Move divergence, held-out specs)
   are standing candidates every audit re-triages.
3. Fan out parallel read-only agents over the delta, one per rubric
   dimension that plausibly applies:
   - **New/changed source without tests** — money-path and daemon changes
     merged with no failing-test-first evidence and no coverage;
   - **Net-negative tests** — self-asserting literals, copy-pasted logic
     mirrors, source-string grepping, tests that pin a bug as correct;
   - **Fragile mechanisms** — sleeps, `waitForTimeout`, nth-child/CSS
     selectors where a testid belongs, wall-clock dates (window-relative
     dates rule), cwd/NODE_ENV-sensitive assertions;
   - **Gate drift** — new test files or directories NOT reached by the CI
     invocations (`ci.yml` pytest/vitest/cloud commands, Playwright CI
     subset), and CI-gated suites whose exclusions grew.
   Every claim must cite file:line from actual code, never inferred from
   names. Scope agents to the diff plus its blast radius, not the tree.
4. Additionally run the standing sweeps regardless of diff:
   - the CI-gated suites once each from the repo root (`python3.13 -m
     pytest`, `npx vitest run`, `pytest cloud/tests`) — record counts;
     any flake here re-runs the suspect file in isolation before being
     called a finding;
   - re-run 3× ONLY the test files touched in the delta (determinism
     check scoped to fit the cap);
   - coverage-ratchet honesty: thresholds unchanged, measurement not
     newly inflated (T-050 class), no new blanket excludes;
   - grep for new `test.skip` / `it.skip` / `pytest.mark.skip` /
     `xfail` introduced in the delta without a linked T-### or issue.
5. Dedupe against ALL existing T-### findings. Append genuinely-new
   findings to `TEST_AUDIT.md` under a dated `## Delta audit <date>`
   section (cite file:line, severity P0/P1/P2, continuing T-numbers) and
   add backlog rows with red/green acceptance criteria. Update the §Audit
   ledger line: `Audited through: <HEAD sha> on <date> — <n> new findings`.
6. Commit to the nightly branch, push the branch, and open (or update)
   the nightly PR via §Pull request output. Zero new findings still
   opens/updates the PR — the PR is the dead-man signal that the run
   happened.

## Mode: remediate (second phase of the daily cycle)

Goal: work EVERY verified un-DONE finding from this cycle's audit in
severity order (P0, then P1, then P2), then older stragglers, exactly by
the PART B contract:

**Remediate mandate.** Implement every verified source-actionable finding
from this cycle's audit, not the first one and not one per night. Group fixes
by root cause into separate commits on one dated branch `testing/<YYYY-MM-DD>` (one
branch per loop per day; the deliver phase turns it into one PR). Red/green
per fix; the full project gates before every commit. Independent fixes may
run in parallel as subagents in separate worktrees of this clone
(`git worktree add ../wt-<id> -b testing/<date>-<id> testing/<date>`), each
committing to its own branch; this phase merges them back onto the dated
branch, reruns the gates on the merged result, and removes the worktrees
(`git worktree remove`, `git branch -d`). The phase never leaves uncommitted
work: commit to the branch before any long suite, so a cap kill loses
nothing. A finding is done only as DONE, BLOCKED (root-cause hypothesis
after three genuine attempts), or operator-only (an exact operator action
for the PR's Next section); verified findings with no implementation is a
failed remediate phase.

1. Check out the nightly branch (create from `origin/main` if the audit
   phase produced nothing; then this run only re-verifies gates, step 4).
2. Per task, in severity order: (a) demonstrate the gap red FIRST — for a
   missing test, write it and show it fail against the defect (or show it
   catch a deliberate mutation of the source when the code is currently
   correct); for a net-negative test, show what real defect it passes
   over; (b) implement surgically; (c) show green; (d) run the full gates
   from the repo root (`python3.13 -m pytest`, `npx vitest run`, and
   `pytest cloud/tests` when units/cloud files changed); when the task
   changed UI, also run the relevant `web/e2e` spec in the worktree and
   attach the screenshot: the clone-copied `node_modules` is what makes
   that possible, and CLAUDE.md does not accept unit-only evidence for
   UI; (e) append the TEST_LOG.md row with red/green counts; (f) commit
   with the T-### id.
   Source-code fixes are in scope ONLY when a test correctly fails
   against a real defect the audit identified — fix the defect, keep the
   test; never the reverse.
3. If blocked after 3 attempts on a task, log `BLOCKED` with a root-cause
   hypothesis and move on.
4. Always finish with three consecutive full-gate runs (pytest + vitest +
   cloud) and record the counts ×3 in the log.
5. Push the branch; rewrite the PR via §Pull request output. DONE/BLOCKED
   tables and gate counts ×3 go on the rolling issue. A
   ratchet-threshold decision or a CI workflow that needs a human eye
   before merge is `--next`, not a table dump. CI on that PR is the deliver
   phase's job (§Mode: deliver).

## Mode: deliver (third phase of the daily cycle)

Goal: every commit the remediate phase landed on `testing/<YYYY-MM-DD>` reaches the
operator as ONE pull request with CI green, in this same cycle, and the
operator is told exactly what is ready to merge. The loop never merges.
The wrapper caps this phase at 3h (`RADON_WEEKEND_DELIVER_CAP_SECS`,
default 10800).

1. Resume first. Read this loop's deliver record
   (`python3.13 scripts/nightly_deliver.py show --loop testing`; kept outside the clone under `~/radon-weekend/.testing-deliver/`).
   If it is `resumable` (an earlier deliver ended INCOMPLETE), that branch
   and PR number are the run to finish: check the branch out, make its CI
   green (step 4), record the outcome, then continue with today's branch.
   Never open a second PR for a branch that already has one.
2. Push the dated branch. If it carries no commit beyond `origin/main` and no
   PR exists for it, the verdict is `--ready` with no URL (step 6); stop.
3. Open ONE PR for the branch via §Pull request output (`--loop testing`);
   update the existing PR when one is already open for the branch (`gh api
   -X PATCH`). Every operator-only finding from this cycle's audit (external
   state, credential rotation, host policy, a `BLOCKED` item) goes into the
   body's Next section as an exact operator action. Nothing is dropped
   silently. Record the PR:
   `python3.13 scripts/nightly_deliver.py record --loop testing --branch <branch> --pr <n> --url <url> --status pending`.
4. Wait for CI, bounded:
   `python3.13 scripts/nightly_deliver.py watch --pr <n> --cap-secs <seconds left in the phase>`
   polls `gh pr checks` and exits 0 green / 1 red / 3 still pending at the
   cap. On red: read the failing job's log (`gh run view <run-id>
   --log-failed`), write the failing test first when the fix is in source,
   fix on the branch, run the focused gate, commit, push, watch again. Repeat
   until green or the cap. Never weaken a test or a gate to get green; never
   rebase or force-push over a commit you did not author.
5. Record the outcome (`record ... --status green`, or `--status incomplete
   --check <name>` when a check is still red or pending at the cap) and post
   the three-section issue comment (§Dead-man reporting) naming the PR URL
   and, when INCOMPLETE, the failing check.
6. Print, as the LAST stdout line of the phase, the verdict line from
   `python3.13 scripts/nightly_deliver.py verdict --loop testing --ready <url>...`
   (or `--incomplete <check> --pr-url <url>`). The wrapper greps it:
   `NIGHTLY DELIVER READY: loop=testing prs=<n> <urls>` becomes the operator
   notification "N PR(s) green, ready to merge: <urls>" (Pushover and the
   dead-man comment); `NIGHTLY DELIVER INCOMPLETE: loop=testing check=<name>
   pr=<url>` becomes "INCOMPLETE: <name>", the phase exits 75, and the next
   fire resumes the same branch and PR from the record. An exit-0 deliver
   phase without the line is INCOMPLETE. Never emit the line anywhere else.

## Declaring a no-op phase

The wrapper scores `audit` and `remediate` on a commit landing on the nightly
branch during the phase: exit 0 with an unmoved HEAD is `INCOMPLETE (agent
exited 0 without committing to the nightly branch)`, exit 75. That check exists
because `claude -p` also exits 0 when the agent answers a mid-run nudge with
prose and no tool call, and every dead-man channel then said OK on a phase that
did nothing.

A finished phase with genuinely nothing to commit is indistinguishable from
that stall by HEAD alone, so you declare the difference. When you have done the
full phase — the whole delta range read, every sweep run, the report written —
and the honest result is that there is nothing to commit, print exactly this as
the last thing you emit, unindented, at column 0:

```
NIGHTLY PHASE NO-OP: loop=testing phase=<audit|remediate> <one-line reason>
```

For example (indented here on purpose — see the third rule below):

```
    NIGHTLY PHASE NO-OP: loop=testing phase=audit no new findings in the delta range
    NIGHTLY PHASE NO-OP: loop=testing phase=remediate 0 source-actionable P0/P1 items
```

Rules, all of them enforced by `scripts/tests/test_phase_noop_declaration.py`:

- The line must name THIS loop and THIS phase. A line copied from a sibling
  loop or a different phase does not count.
- It must start at column 0. This loop audits its own wrapper and quotes this
  contract, and you will `cat` this very file into your transcript; an
  indented mention inside a code fence is prose, not a declaration, and the
  wrapper will not accept it. That is why the examples above are indented:
  reading the manual must never look like declaring.
- It is a declaration of completion, not an excuse. Emit it only when the phase
  ran end to end. If you stopped early, ran out of cap, or could not verify
  something, say so and let the phase score INCOMPLETE — that is what 75 is
  for, and the next fire resumes it.
- Never emit it when you did commit. A commit is its own evidence.
- Silence is still INCOMPLETE. Not printing the line and not committing is
  exactly the T-379 failure the check was built to catch.

## Long stages run detached and are awaited in-session

A phase never returns while a stage it started is still running. "Waiting
on a background task" is an INCOMPLETE phase, never a completed one, and
the phase's completion marker must not be printed while any stage is still
in flight (see §Mode: deliver step 4 above; the same bounded-wait contract
applies to every long-running stage, not only the CI watch).

Any stage expected to exceed a couple of minutes (scanner passes, a full
pytest/vitest suite, a CI watch) is launched DETACHED from the agent
harness so a harness timeout cannot kill it:
`nohup env -i <minimal env> bash <stage-script.sh> </dev/null >stage.out
2>&1 & disown` (macOS has no `setsid`). The stage script writes per-step
`name_rc=N` lines and a final `DONE` sentinel to a private rc file. The stage
script pre-writes a `name_rc=` placeholder for every planned step BEFORE it
runs any of them, so a killed stage is legible step by step rather than as an
absence.

**An rc file with no `DONE` is a FAILED stage, never a passing one.** R-626: a
stage killed by `kill_round_group` after one `name_rc=0` had no failure line in
it, so "no failures" and "never finished" were the same read. Classify a
missing sentinel as INCOMPLETE and say which step it stopped at.

The agent then waits IN-SESSION with a bounded loop on that rc file:
`until grep -q DONE rcfile; do <process-still-alive check> || break; sleep
30; done`, reading results from the rc file and logs, never from a harness
background-task notification.

Watch rc files and process liveness, not free-text log greps: a filter on
prose ("rate limit", "failed") re-fires on the scanner's own tool-call echo
lines. Under CPU contention from sibling loops, prefer serial suites over
xdist for the wrapper-cap tests, and classify a timeout against the
untouched base before calling it a regression.

## Pull request output

PR titles and bodies are generated by `python3.13 scripts/github_pr_output.py`,
never freehanded. Pass `--loop testing`, `--date`, `--issue` (what went
wrong, as one bullet per finding: `- **Component**: what happened.`), `--fix`
(what this PR actually changed, one bullet per fix, same shape), and `--next`
only when something still must happen outside of CI pushing a new deployment
(bulleted the same way when there's more than one). Omit `--next` and the
formatter emits `Fixed with green deployment`. A single plain sentence still
works when there is exactly one finding.

The body has exactly three sections, in this order: **Issue discovered**,
**What was done to fix it**, **Next**. Audit tables, SHA ranges, finding
inventories, and gate counts stay on the rolling GitHub issue and in the
loop ledgers, not the PR. Title shape: `Testing <date>: <plain-language
issue>`. Create a new dated branch, or a new remediation PR after the
audit PR merged, with `gh pr create --title <title> --body <body>
--head <branch> --base main` (or `POST /repos/{owner}/{repo}/pulls` with
`head`, `base`, `title`, and `body`). Formatter `--json` is `{title, body}`
only; do not POST it as the create payload. Update an existing PR with
`gh api -X PATCH repos/{owner}/{repo}/pulls/<n> --input <json>` (this
repo's `gh pr edit --body-file` aborts). Verify with a grep for a phrase
you just wrote.

Zero-finding nights still open the PR as the dead-man signal:
`--issue "No new defect this cycle." --fix "Recorded the audit. No code change." --next "No deploy needed."`

## Dead-man reporting

Every phase outcome is reported three ways, so a silent-dead runner shows up
the next morning at the latest: a comment on the rolling GitHub issue
labeled `testing-nightly`, a Pushover notification per phase carrying the
status and the nightly PR link when one exists, and the PR itself.

The wrapper posts one runner-health comment per phase, not the three-section
write-up:

**PHASE** STAMP **status**
optional detail

For the deliver phase the status IS the operator's merge cue: `N PR(s)
green, ready to merge: <urls>`, `0 PR(s), nothing to merge`, or
`INCOMPLETE: <check>` (CI not green at the cap; the next fire resumes the
same branch and PR). The issue is created once with a timeless
rolling-dead-man description. Run
history stays in comments. The wrapper does not edit the issue body after
the first run. A missing daily comment means the runner did not fire.

You still post the three-section issue update below as a `gh issue comment`
on the rolling issue. Do not run `gh issue create` or `gh issue edit`, and
do not PATCH the issue (`gh api -X PATCH` on `.../issues/`). That would
overwrite the dead-man description. Comment-only. The wrapper also comments;
you are not the only commenter. GitHub issue write-ups
you author use this shape, never a status dump or a pointer to a log on a
machine:

**Issue discovered**
What went wrong, in plain language. If nothing went wrong, say that.

**What was done to fix it**
What THIS run actually changed. If nothing: "Nothing this run."

**Next**
Only work that must happen OUTSIDE of CI pushing a new deployment. If
nothing remains: "Fixed with green deployment"

`INCOMPLETE (agent exited 0 without committing to the nightly branch)` is
the status the wrapper posts when `claude -p` returned 0 but no commit landed
on the nightly branch during the phase (T-379): treat it exactly like
TRUNCATED — the phase's draft work, if any, is under `/tmp/tw-<date>/` and
the next phase must land it. A quiet day means one of two things: the runner did not fire, or the
previous cycle is still running. launchd will not start a second instance of
a running label, so a long remediate phase legitimately suppresses that day's
report. Check `launchctl list | grep radon` before treating quiet as dead.
This loop's worst case is one 2h audit plus one 6h remediate plus one 3h
deliver, so it always clears the next 00:00 fire.

## Measure improvement

Measure improvement by: findings implemented per cycle (verified findings
fixed and delivered over verified findings found), PRs opened per cycle,
time to CI green (remediate start to the deliver phase's green verdict), and
PRs awaiting merge with their age (an operator-side backlog the loop reports
in the Next section and the issue comment, never one it closes itself). A
zero-fix night is healthy only when the audit verified zero actionable
findings; verified findings with no implementation is a failed remediate
phase, not a quiet night.

## Self-improvement

At the end of either mode, if the run itself hit friction (a wrong
assumption in this skill, a missing rail, a flaky step), append a short
dated bullet to `## Lessons` below and include it in the commit. That is
how this loop improves as the codebase grows.

## Lessons

- **2026-08-16 (audit):** start by checking the runner clone is CLEAN, before
  anything else. This run opened on orphaned WIP from a prior capped run —
  three modified files plus an untracked test importing a module that does not
  exist — which aborted pytest at COLLECTION (exit 2, zero tests run) and would
  have been misread as a red gate. Park it recoverably
  (`git stash push --include-untracked -m "<loop>-<date>: parked ..."`), never
  discard it, never commit it, and record the stash ref in the audit so the
  operator can recover it. Gate counts are only meaningful from a clean tree.
- **2026-08-16 (audit):** do NOT run the determinism re-runs concurrently with
  each other. Running vitest ×2 alongside pytest ×2 on one machine is what
  surfaced T-062 — which was genuinely useful, but the skill's rule ("re-run
  the suspect file in isolation before calling it a finding") is what separated
  the real race from contention noise. Run the gates serially for the counts,
  then deliberately re-run under load if you want to hunt races.
- **2026-08-16 (audit):** the "re-run 3× ONLY the delta-touched test files"
  rule does not scale to a week-sized delta. This one touched 263 of
  `web/tests` and 100 of `scripts/tests` — effectively the whole suite — so
  scoped re-runs collapsed into full-gate runs. Say so in the audit rather than
  pretending the scoping happened.
- **2026-08-16 (audit):** `rg` on this runner resolves to BSD `grep` (no
  `--glob`, no `-N`), and the rtk proxy mangles piped `grep` output. For any
  non-trivial scan of a large diff, write the patch to a file and parse it with
  a `python3.13` heredoc — that is what produced the trustworthy zero-new-skips
  result.
- **2026-08-16 (remediate) — NEVER run `git pull` on this runner, and push
  after EVERY task commit.** The rtk hook rewrites bare `git` commands, and its
  `git pull` rewrite did `reset --hard origin/<branch>` + a rebase onto
  `origin/main` while printing "Already up to date." That silently discarded 14
  unpushed remediation commits (T-055…T-069 — 29 files, +1296 lines). The same
  filter then served STALE `git log` / `git status` output, so the loss stayed
  invisible for several tool calls; it surfaced only because a baseline vitest
  run reported a `.pi` suite that T-058 had already fixed. Rails:
  - Use `rtk proxy git …` for every git command in this loop. Bare `git`
    output on this runner cannot be trusted for state decisions.
  - Never `git pull`. Sync with `rtk proxy git fetch origin` plus an explicit
    `merge --ff-only` you chose deliberately.
  - `rtk proxy git push` the nightly branch immediately after every task
    commit, not once at the end. The branch on origin is the only durable copy;
    "push at the end of the run" is a single point of failure.
  - Recovery if it happens anyway: `rtk proxy git reflog` still holds the
    orphaned tip. Tag it first, then
    `git rebase --onto <rebased-base> <old-base> <orphan-tip>` and
    `git branch -f`.

- **2026-08-17 (remediate): never pipe a gate run through `tail` alone.**
  Round 1 of the closing 3x gate reported `10 failed | 6706 passed` and the
  names were gone — the command kept only the summary line. Seven further
  full runs (four sequential, two deliberately concurrent with a full
  pytest) were all `6716 passed`, so the round could not be named or
  reproduced and had to be logged as an observation rather than a finding.
  Write the full reporter output to a file per gate run and read the tail
  from that file, so a flake round is nameable the first time it happens.
- **2026-08-17 (remediate): `pytest cloud/tests` is red on macOS on
  `origin/main` too.** Diff the failure LIST against a clean `origin/main`
  worktree before treating any cloud red as yours; the count alone is not a
  signal. Baseline as of this run: `10 failed, 848 passed, 4 skipped`, then
  attributed to `sha256sum`. **That attribution is STALE as of 2026-08-29**
  — `/opt/homebrew/bin/sha256sum` exists on this host and no `sha256sum`
  red remains. The current darwin baseline is `37 failed`: 13 in
  `test_bootstrap_control_plane.py` (`exec {fd}<>` is bash 4+ and
  `/bin/bash` here is 3.2, so it exits 127), 21 in
  `test_ib_gateway_control.py` (`operator-radon.sh` uses `mapfile`, bash
  4+), 3 in `test_caddy_edge_timeouts.py` (no `caddy` on PATH).
  `setup_testing_weekend.sh` now checks both and names the consequence;
  installing either MOVES this baseline, so re-record the FAILED list in
  the same run.
- **2026-08-17 (remediate): pre-flight a spec under `next start` before
  curating it into CI.** The e2e job builds and serves a production
  server, and this repo has a documented dev-vs-prod divergence. Every
  spec added to the curated list this run was verified under
  `PLAYWRIGHT_WEBSERVER_CMD="npx next start"`, which is also how
  `performance-twr-payload.spec.ts` was caught as permanently red before
  it could red the job.
- **2026-08-22 (audit): last weekend's remediation lands inside this week's
  delta.** The ledger SHA is the audit HEAD, not the merge of the nightly PR,
  so the range `71de8a33..HEAD` re-contained T-055…T-079 and the reliability
  loop's REL-0xx source commits. Re-triage them as ordinary delta rather than
  exempting them (two findings this run — T-086, T-087 — were on REL-038
  tests), and say in the audit that the range overlaps.
- **2026-08-22 (audit): the darwin cloud baseline is a LIST, not a count, and
  it moves.** Round 1 read `12 failed` against a recorded baseline of 10; the
  diff of `FAILED` lines against the 2026-08-17 list is what separated two
  new environment-shim reds (T-088) from the known ten. (The environment
  cause has since changed from `sha256sum` to bash 3.2 + missing `caddy`;
  the list is 37 today.) Always `sort` the
  `FAILED` lines to a file and `diff` them; update the recorded baseline in
  the audit whenever it changes.
- **2026-08-22 (audit, second pass): CHECK `origin` FOR AN EXISTING WEEKEND
  BRANCH BEFORE YOU START, not at push time.** Two runs of this loop audited
  the same range on the same day on different hosts. The second only
  discovered the first when `git push` was rejected — after it had already
  numbered 32 findings from T-080, colliding with all 17 the first had
  pushed. Do this in step 1, right after the clean-tree check:
  `git fetch origin && git rev-parse --verify origin/testing/<date>`.
  If it exists, read its audit section FIRST and continue numbering after it.
  Recovery if you find out late: never force-push over the other run. Reset
  onto its tip, drop your duplicates, renumber the rest from its highest
  T-number, and append a `## Delta audit <date> (second pass)` section — the
  frozen-contract rail permits a new dated section, never a rewrite. Record
  the convergences in a table; two independent readers landing on the same
  file:line is real evidence, and throwing it away is a loss.
- **2026-08-22 (audit): cross-references written into the sweeps / re-triage
  prose go STALE while you are still drafting.** Three references in the
  first draft ("Filed as T-096", "Promoted to T-095", "see T-094") were
  written against early draft numbers and silently pointed at three unrelated
  findings by the time the section was numbered. Number the findings FIRST,
  then write the prose that cites them — or grep every `T-\d{3}` in the
  finished section and confirm each one resolves to the subject you meant.
- **2026-08-22 (audit): verify the RUNNER TOOLCHAIN before trusting a red
  gate, the same way you verify the tree is clean.** One round reported
  `107 failed` and every failure was "async def functions are not natively
  supported" — the venv (then the shared `~/radon-weekend/venv`; now this
  loop's `~/radon-weekend/venv-testing`) had pytest but no
  `pytest-asyncio`, which only CI installs. The same tree was `7216 passed`
  once the plugin was in. `node` was also absent from the agent's PATH until
  `~/.nvm/versions/node/<v>/bin` was prepended (the wrapper exports it, but a
  Bash-tool shell re-reads the profile). Do this before the gates:
  `python3.13 -c "import pytest_asyncio"`, `node --version`,
  `ls node_modules/.bin/vitest`. Fix the environment, never the repo, and
  record the install in the audit.
- **2026-08-22 (audit): attribute a red cloud gate by RUNNING the base SHA.**
  Building on the first pass's "baseline is a LIST, not a count": a
  `git worktree add /tmp/... <last-audited-sha>` plus a `diff` of the sorted
  `FAILED` lists settles it in two minutes and byte-identically, and it also
  catches the case where the list is longer for a reason unrelated to the
  delta — this host reads 34, not 12, because it has no bash >= 4.
- **2026-08-22 (audit): a source change can make an UNTOUCHED test
  date-dependent — sweep the diff's blast radius, not the diff.**
  `f2fbe0a7`/`d45849d7` added an `isIbDailyPnlCurrent()` wall-clock gate to
  `MetricCards`; two e2e specs the delta never opened now false-red every
  weekend (T-117). Nothing in the changed-test list would have surfaced it.
  After cataloguing changed tests, ask the inverse question: which EXISTING
  tests does this source change now describe differently?
- **2026-08-23 (remediate): an absent audit phase does NOT mean "only
  re-verify gates".** This cycle's audit never ran (PR #75 had merged the
  2026-08-22 findings at 11:17 and no 2026-08-23 branch existed), but the
  backlog still held 20 un-DONE P1s from T-081…T-109. Step 1's "create from
  `origin/main`, then only re-verify" applies when the backlog is EMPTY;
  otherwise create the branch and work the newest non-P2 stragglers exactly as
  if this run's audit had filed them. The remediation bullets go under a
  `## Remediation <date>` section in `TEST_AUDIT.md` and a dated table in
  `TEST_LOG.md`.
- **2026-08-23 (remediate): fan the backlog out to one worktree per task
  group; cherry-pick back serially.** `git worktree add --detach /tmp/...`
  plus an APFS clone copy of `node_modules` (`cp -Rc <clone>/node_modules
  <wt>/node_modules` and the same for `web/`; fall back to `cp -R` off APFS)
  gives each subagent a clean tree; the loop venv needs nothing. NEVER
  symlink `node_modules`: a symlink out of the worktree root breaks BOTH
  gates. vitest cannot resolve `@rollup/rollup-darwin-arm64` through the
  link's real path, and Turbopack hard-fails the Playwright webServer with
  `Symlink [project]/web/node_modules is invalid, it points out of the
  filesystem root`, so the worktree cannot run e2e at all. `cp -Rc` is
  copy-on-write, so it costs seconds and near-zero disk. Group findings that
  touch the SAME test file into one agent (T-082+T-097, T-084+T-099,
  T-086+T-098 here) or the cherry-picks conflict. The main clone stays
  untouched, so a baseline gate can run there while the agents work, and
  each `cherry-pick -n` + docs row + push is one durable commit. Sixteen
  P1s landed in ~15 minutes of wall clock this way versus one-at-a-time.
- **2026-08-23 (remediate): a subagent's "green" is scoped; re-read the
  source diff before landing.** Two things the per-task reports could not
  show: (a) the relay is ESM with socket side effects on import, so T-087's
  builder was never executed by the relay in any test — verify by hand that
  the variables the extracted call uses (`freshness`) are in scope at the
  call site; (b) `_read_deploy_evidence` gained a `now` kwarg (T-103) and its
  second caller lived in `grok_page_responder.py`, outside the agent's
  scoped run. Grep every caller of a changed signature in the LANDED tree,
  not the worktree.
- **2026-08-23 (remediate): the darwin cloud baseline grew by three
  environment-class reds without any test being wrong.** At `4985a7f8`
  this host reads 12; at `2e904678` it reads 15 because
  `test_refresh_control_plane.py` was new in the delta. (The `sha256sum`
  cause named at the time is stale; as of 2026-08-29 the baseline is 37 and
  the cause is bash 3.2 + missing `caddy`.) Same rule as the audit lesson: sort the
  `FAILED` lines, run the base SHA in a worktree, `diff` — and record the
  new list in the log so the next run does not misattribute it.
- **2026-08-23 (remediate): two hosts remediated the same branch at once —
  the 2026-08-22 "check origin first" lesson is necessary but not
  sufficient for REMEDIATE.** Both runs fetched at pre-flight, found no
  branch, and created it; the second host's first push was rejected, it
  reset onto this host's tip (correctly) and started from the BOTTOM of the
  P1 list — which this host had already fanned out in parallel, so T-100,
  T-106, T-108, T-109 were still at risk of being done twice. Rails:
  push the EMPTY branch immediately after creating it (this host did, and
  that is what made the second host detect the collision); before EVERY
  landing, `rtk proxy git fetch origin` and rebase onto the remote tip with
  `rtk proxy git rebase` (never force-push, never `git pull`); keep the
  per-task landing script inserting rows ABOVE any other host's section
  in `TEST_LOG.md` so the two tables do not interleave; and list every
  landed T-### in the PR body as soon as it lands, because the PR body is
  the only channel the other host reads. A `TEST_LOG.md` conflict on
  rebase is expected; resolve it by keeping both sections, never by
  dropping a row.
- **2026-08-23 (remediate, second host): a CLAIM COMMENT on the PR is what
  actually de-conflicts two live runs.** The "check origin first" rail did not
  fire here — `origin/testing/weekend-2026-08-23` did not exist at pre-flight
  and appeared before the first push — so the first two tasks (T-081, T-109)
  were done twice and thrown away. What stopped it was posting a comment on
  the nightly PR naming the exact T-### items this host would take, BEFORE
  starting them; zero collisions across the five that followed. Do it as soon
  as the branch exists: list the items, say which end of the list you are
  working from, and re-`fetch` before every landing.
- **2026-08-23 (remediate, second host): a duplicate task is not wasted if you
  DIFF the two answers.** Both hosts fixed T-081; comparing the two
  implementations is what found that the landed one keys precedence on
  `report_date` alone, which drops a second account's mirror-only row.
  Reset onto the other host's tip, drop your commit, then probe their fix with
  YOUR test cases before moving on. That was the only surviving product change
  from this host's first hour.
- **2026-08-23 (remediate): `rtk` is not installed on every runner.** The
  2026-08-16 lesson mandates `rtk proxy git …`; on this host `rtk` is not on
  PATH at all and bare `git` is correct and trustworthy. Check
  `command -v rtk` at pre-flight and follow that rail only where the proxy
  actually exists — otherwise every git call fails with exit 127 and the run
  looks blocked.
- **2026-08-23 (remediate): check `uptime` before calling a vitest round red.**
  One full gate read `13 failed / 7169 passed`, 11 of them bare
  `Test timed out in 5000ms` across nine unrelated files, with the run taking
  336 s instead of ~110 s. Load average was 66 (`corespotlightd` at 367% CPU).
  The nine files were 44-passed in isolation and the next full run was
  7182 passed in 107 s. Capture the reporter output to a file, name the files,
  re-run them in isolation, and record the load average alongside the counts.
- **2026-08-23 (remediate): a new tree-walking contract test must be timed, not
  just made green.** The first draft of the inverted table-overflow contract
  built a fresh RegExp against the whole ~1 MB `globals.css` for every class
  token of every ancestor of every tag: 4.5-6 s against vitest's 5 s default,
  so it flaked 8/8 on a TIMEOUT rather than an assertion. Precompute the
  stylesheet side once and re-run the new file 3x checking the reported
  duration, not only the pass count.
- **2026-08-23 (remediate): inverting a net-negative contract surfaces PRODUCT
  defects — budget for filing them, not for fixing them.** Turning the
  table-wrapper test from "named wrappers must be styled" into "every table
  must have an overflow ancestor" produced six real horizontal-overflow bugs
  (T-121). Fixing them is six UI changes needing 390px browser verification,
  which is outside a test-quality task. Pin them in a named list under an
  EQUALITY assertion — so a seventh reds immediately and fixing one reds until
  its entry is removed — and file the finding. Do NOT skip them, and do NOT
  quietly widen the rule until they pass.
- **2026-08-25 (audit): when CI's test invocation changes shape, diff COLLECTION, not
  pass counts.** `424e66da` sharded pytest into shell globs (`test_[a-c]*.py`) that
  cannot match a directory; CI stayed green while 752 tests in two subdirectories
  stopped running. `pytest --collect-only -q` over the full tree vs the union of the
  CI path sets, `sort -u` on the file names, `comm -23` — two minutes, and it is
  the only check that sees a silent drop. Pull the per-job pass counts from CI
  (`gh api repos/{owner}/{repo}/actions/jobs/<id>/logs`; `gh run view --job --log`
  returns empty on this host) and compare the shard SUM to the last unsharded run.
- **2026-08-25 (audit): agent-reported findings need a lead spot-check before
  filing, and it is cheap.** Six agents returned ~45 candidates; every one the
  lead re-read at the cited line held, but two summaries overstated a mechanism
  (an `apply_` stamping claim that needs a `contract.secType` the agent's repro
  omitted). Reproduce the top P1 in-process from the cited file (a 10-line
  python heredoc), read the cited lines of every P0/P1, and only then number.
- **2026-08-25 (remediate): this runner has no `rtk`, no `setsid`, and no
  `pytest-xdist`.** Bare `git` is the only git here and its output was
  trustworthy (the 2026-08-16 rtk lesson applies only where rtk is installed —
  check `which rtk` first). `pytest-xdist` is CI-only like `pytest-asyncio`
  was: install it in this loop's venv (`~/radon-weekend/venv-testing` — the
  legacy shared `~/radon-weekend/venv` is unused since the per-loop split)
  before verifying anything under
  `-n auto --dist loadfile` (a new shard is only proven with CI's flags).
  `setsid` does not exist on darwin: detach a long job with
  `subprocess.Popen(..., start_new_session=True)`, never `nohup setsid`.
- **202

…(truncated)
