Orchestration

Orchestration

Delegation only works when the orchestrator treats every subagent claim as unverified and every judgment call as its own. These rules were each paid for by a real incident.

Charters

A charter is the task prompt a subagent receives. Requirements:

• Reference every document by absolute path — a relative path resolves differently (or not at all) in the agent's working directory.
• When the work depends on an external spec or vendor document, fetch it raw — curl or a real browser, never a summarizing fetch — into a scratchpad cache file, and name that cache path in every dependent charter. All lanes then build from identical bytes; left to fetch it themselves, each agent works from its own paraphrase and the lanes drift apart on the contract they were supposed to share.
• When parallel lanes could touch the same module, name the sibling lanes and give each an explicit do-not-touch list of files/functions the others own. For genuinely shared append-only files (routes config, nav layout, seeds), say up front that rebase conflicts there resolve keep-both. Before launching, check each item's designed fix against its lane's do-not-touch list: an item whose only defensible fix lives in a sibling-owned file moves to the lane that owns the file, or gets an explicit carve-out — left as is, the builder must choose between skipping the item and violating the boundary, and one wave produced a skipped item, a banked residual, and a flagged two-file deviation from exactly this.
• When parallel lanes allocate from a shared identifier pool — seed namespace prefixes, fixture ids, ports, queue names — the orchestrator publishes one canon allocation table in every lane's charter before the first launch, and each lane sub-partitions its allotment across its own flows. Lanes that pick identifiers independently collide, and the collisions surface as another lane's spec failing long after both lanes went green in isolation; one wave shipped three separate collisions (two lanes sharing a phone prefix, two sharing a fixture id, one lane colliding with itself) before the table became standing practice.
• When several lanes will each touch a shared computed constant — a contract fingerprint, a generated schema hash — every charter declares the same resolution protocol up front: on rebase, recompute the value by the mechanism that owns it (its own test's computation), never keep-mine or keep-theirs. Three parallel lanes once rewrote the same fingerprint off stale bases, and every merge needed a hand repin. Each charter also declares whether its own items are expected to move the constant; an unexpected movement is a stop condition to report, never a silent repin — a lane that repins a public contract hash it had no business moving ships the change invisibly under a green suite.
• Assign each subagent an explicit scratch-file namespace — its lane or task name as a filename prefix — and forbid it from writing any shared scratchpad file, above all the orchestrator's session ledger, whose actual path the charter must give since it is named for the effort rather than by a fixed convention. Parallel agents that share a scratch filename clobber each other's state, and a fix agent that writes its own notes over the session ledger destroys the orchestrator's recovery record.
• Include a stop-on-contradiction clause: if the charter's factual premise (a claimed bug, a claimed missing feature, a spec citation) doesn't survive verification against the primary source, the agent halts that item and reports with citations instead of inventing. "Verified already correct, no change made" is a valid, expected outcome. The clause covers the charter's designed fix shape and its measured file:line anchors, not only the finding: sibling merges stale both between charter-writing and execution, so a builder that re-verifies the design against its own worktree and deviates with citations is complying — the deviation is adjudicated at merge, never treated as disobedience.
• A charter touching product-monolith instructs the agent to read product-monolith/docs/process/engineering-conventions.md (absolute path in the lane's worktree) before coding and to treat it as binding, naming the repo's designated reference app as the reference implementation; a reviewer or skeptic charter for product-monolith names that file as the codified-house-rule source for rubric (b) below. A charter that writes or reshapes tests also names docs/process/testing.md as binding and does not prescribe a test mechanic itself — a charter once offered a patch-based interleave that file forbids, and only the builder's own diligence avoided shipping it.
• A charter that includes a mutation proof (deliberately breaking code to show a test goes red) also specifies the restore protocol: restore byte-identically, verify via git status/git diff, and purge __pycache__ under the touched packages — a same-size restore within the same mtime second leaves stale bytecode that quietly poisons every later run in that tree.
• When two reference sources disagree, the charter names the conflict for adjudication — it never silently picks one. A charter's factual claims are no more trustworthy than a builder's until verified.
• Builders never run long gates. A charter ends at commit (or commit + push) and STOP, with only scoped foreground checks before it (the touched module's tests, lint) — "run the full suite in the foreground" does not survive contact with builders: they background it and end their turn terminally stalled, because a workflow agent's turn ending is final. The orchestrator runs full gates itself as its own background shell (its re-invocation on completion costs nothing) and owns push/PR/merge when the charter stops at commit.
• Write each new charter/script fresh with the Write tool. Deriving one by programmatically splicing a previous one introduces silent syntax errors.
• Relay a finding at exactly the scope it was measured. "These two named tests survive the mutation" is not "the suite survives the mutation"; widening it on the way into a charter sends the agent to disprove a claim nobody made, and its correction can bury the narrower gap that was real. Quote the finding's own boundary, and mark anything you extrapolated as yours to verify.
• Cross-check a charter's exit assertions against its own items before launching: an exit gate demanding a file show no diff contradicts another item that edits the same file, leaving the agent to choose between violating the gate and dropping the item. Scope byte-identity checks to the exact lines or symbols they protect, never to whole files other items touch.
• A parameterized script must parse args defensively (string-or-object) and hard-throw on any missing required field — a silently-undefined interpolation produces a plausible-looking empty result that reads as a clean pass.
• A slice that ships a route, loader, or form must drive that surface live at least once — a minimal browser interaction or a real-Request smoke — before handoff. A unit-only exit bar passes code whose query crashes at runtime or whose form cannot submit (a select with no name attribute).
• A skeptic or verifier charter carries TWO acceptance rubrics, not one: a finding stands if either (a) its concrete-input failure scenario occurs at runtime, or (b) it identifies a violation of a codified house rule or doctrine in the work under review. A runtime-only rubric refutes correct doctrine findings — an ad hoc query fragment inlined in a view where product-monolith/docs/process/engineering-conventions.md prescribes a model-owned queryset method — exactly when the fix is cheapest. Doctrine findings against unmerged schema or shape default to fix-now, because schema shapes are effectively permanent once merged.

Verify, never trust

• Re-run the gates yourself before advancing any stage on a subagent's self-report.
• A completeness claim — "every route covered", "every call site migrated", "every consumer updated" — is only checkable against a denominator derived live from the system of record (the URL resolver, the schema, a repo-wide census), never against the plan that produced the work: the plan omits exactly what the work omitted. Establish the denominator before declaring any "every X" objective met, and prefer a gate that recomputes it and fails on drift over a registered list or document describing coverage — registered state duplicates the system and goes stale silently.
• Reconcile numbers exactly: baseline + computed delta = measured, and identical across two runs. A mismatch is a hard stop, not noise. A collect-only reconciliation carries one extra term: registry-parametrized suites expand on their own whenever the registry they parametrize over grows, so account for that expansion before reading a collect-count jump as drift.
• Reconcile a builder's charter item by item against its diff, not only by counts: every charter item ends in exactly one visible state — implemented in the diff, verified-already-correct, or an explicitly reported skip. Builders silently drop items, and numeric reconciliation cannot catch an item whose omission changes no counts (a test reshape, a mechanical conversion) — one such drop shipped unnoticed and only surfaced in a later personal read of the merged diff.
• Reconcile the adjudicated rulings against the union of the lane charters before the wave launches: every ruled fix appears in exactly one lane's items. Orchestrators drop whole rulings the same way builders drop charter items — three ruled fixes once reached no lane at all, caught only by a ruling-by-ruling reconciliation during the final write-up.
• A charter's expected gate numbers are measured on the current base tip at charter-writing time, never carried forward from per-lane history — every sibling merge stales them. When lanes have merged since the last measurement, re-measure the union before chartering anything that asserts a count.
• A task counts as launched only when its task/run id from the tool result is in the ledger. A written script or a narrated intent is not a running task.
• Never block on a running subagent or workflow — a synchronous wait for its output freezes the conversation until the agent finishes, so the user can neither interject nor redirect, and the user has had to interrupt one to say so. Launch in the background, keep the chat free for whatever else is pending, and act when the completion notification arrives. When the harness's task registration is uncertain (after a compaction), arm a background watcher on the task's own artifacts rather than polling in the foreground.
• Independent lanes launch simultaneously, never in staggered waves. Host contention (CPU, database) slows every lane's gates but never justifies holding a launch back — an idle lane costs more than a slow gate. Contention is managed inside the running lanes instead: full test suites queue under the concurrency cap the worktrees skill sets.
• A review or QA returning zero findings is only a clean pass if its agents all completed. Errored finders, an implausibly fast run, or a wrong agent count mean the run is broken — relaunch it.
• Findings about uncommitted working-tree state from a parallel review fan-out are suspect: when finders share one worktree, one finder's in-flight mutation-proof — code deliberately broken to validate a test, then restored — is visible to its siblings and reports as a phantom defect. Verify any working-tree observation against the commit under review (git show HEAD:file), and either require finders to mutation-prove only against committed state and restore immediately, or discount working-tree-only observations at the verify stage.
• Independently probe a shipping agent's claimed git state (git merge-base <lane> origin/main vs origin/main's tip) before merging. Verify mechanical batch edits landed by grepping the expected before/after state, not by exit code.
• When a merged lane retires or renames a shared symbol (a manager method moved behind a facade, a renamed helper), every sibling lane still to merge can carry code written against the old idiom — the siblings branched before the rename landed, and their files never conflict, so the merge itself flags nothing. At each subsequent sibling merge, grep the whole tree for the retired idiom before running suites and retarget the hits; one seam retirement broke four consecutive lane merges in a single wave, each discovered late by a failing suite or lint instead of cheaply by the grep.
• The orchestrator's own edits are subagent work for verification purposes: any orchestrator-authored change to an audited artifact — a coverage claim, a documented behavior, a spec or manifest sentence — gets the same adversarial re-verification as a builder's diff before it ships. Three separate single-sentence orchestrator edits in one goal each introduced a fresh overclaim that only a verifier caught. Write such sentences from the rendered screen or the measured number, never from the data's intent, then have a verifier try to refute them.
• Personally source-read any confirmed "inconsistent with spec/reference" finding before directing the fix — especially when the fix would reverse a prior ruling. A finding's observation can be accurate while its verdict of "wrong" is not.
• An agent's claim that a symbol has a single consumer ("only seed.ts imports this") is a factual claim — grep the whole repo, tests/ included, before directing a refactor that removes or narrows a shared contract on its strength. It bites hardest when the claim justifies reversing another agent's deliberate earlier design, and green CI is no safety net when the broken consumer only runs against state CI never exercises, like a converging seed over a persistent local database.
• Audit PR-body citations ("per spec X", "per source Y at file:line") against the actual cited location, and check any "deferred as polish / out of scope" claim against the actual contract. Builders produce plausible but fabricated attributions.
• Shipped prose that makes factual claims about system behavior — docs pages, schema and endpoint descriptions, recipe text — is verified like code: before it ships, a dedicated adversarial pass tries to refute each claim against the source. Plausible-but-false wording reads well precisely because it flatters the mental model that wrote it — false revenue-semantics claims survived a builder's self-review and the orchestrator's personal read twice in one PR, and both times only a refute-charter caught them. The unwinnable class is census, totality, and lineage claims — counts of routes or tests, "every caller", "all N migrations", "unchanged from main" — over any surface too large to recount mechanically: seven such claims fell to refute passes in a single review effort, the orchestrator's own among them. Write the mechanism plus verified examples instead, and reserve exhaustive claims for small sets the prose itself defines and a refute pass can recount.
• When a fix pass restructures a seam beyond what its charter asked — replacing an inherited implementation, moving a boundary, collapsing two code paths into one — its acceptance gate is re-running the measurement that exposed the original defect, never the agent's own argument that the new shape is equivalent. The restructure invalidates the earlier measurement, so the evidence you already hold no longer covers what ships. Mandating that re-run in the fix charter caught a second regression the fix pass had introduced while reasoning its way to a correct-looking shape.
• Any change to a shared core module made by a fix/QA agent (core/ in product-monolith, shared_common/ in integration-lambdas, src/core/ in engine-fork) gets the orchestrator's personal diff read before shipping — its blast radius exceeds any lane's review scope.
• When a builder reports it verified a UI flow by simulating requests (fetching the form's FormData, forging a session) instead of real interaction, treat that as an unverified claim and dispatch a real-interaction prober before relying on it. A "harness quirk" explanation for why real clicks failed is itself a claim to verify.

Judgment stays at the top

• Design and adjudication are orchestrator-tier work. When the evidence under a design changes (stale reference, corrected spec), redo the design at the orchestrator tier — never hand the stale design down with a "re-verify your citations" instruction.
• When a lane empirically disproves a factual premise in a shared research digest, correct the digest itself immediately and visibly, before the next lane reads it. Sibling lanes read the artifact, not the report that refuted it, so every later lane inherits the same wrong premise from a document that still asserts it.
• Grounding priority when state or policy is uncertain: primary sources (git, live files, the actual spec) beat live skills, and live skills beat ledger notes or compacted memory. On a policy detail, the live skill file always wins over ledger shorthand.
• When parallel lanes independently invent shapes for the same shared surface, adjudicate one canon and give later-shipping lanes an explicit adopt-the-canon duty at rebase. Distribute the canon as a verbatim, byte-identical file package every lane applies unchanged — identical bytes auto-merge everywhere, so the shared file never conflicts again.
• When lanes re-express existing behavior (authorization gates, validation rules), builders guess wrong in both directions — stricter and looser both shipped from the same wave. Commission one audited map of every capability to its exact source-of-truth behavior (with file:line citations) before any fix pass, and make every fix charter cite the map, never intuition.
• Bank out-of-scope findings (with repro evidence) into a running list for a dedicated pass — never fix them inline, never drop them. Before deferring an item until some future event, check that the event does not itself depend on the item: work parked until a readiness decision it is a precondition for never restarts, because the decision waits on the work and the work waits on the decision.
• A designer's "too invasive / out of scope" blast-radius claim is a factual claim, not a judgment call — grep it before accepting the compromise it justifies.
• Whether to fix a known defect or a regression is not the user's decision. Once verification has established that something is broken, fixing it is the only defensible answer, so asking spends the user's attention on a question with one outcome and stalls the work until they happen to be present. Bring the user a genuine choice between shippable alternatives, or a trade-off only they can price — never a request for permission to remove a defect.
• Before asking the user to rule on a surface or planning a change to it, check the repo for in-flight work: gh pr list plus a scan of remote branches. An open PR may have already settled the question — a dependency queued for a user decision had already been removed by a PR in flight, and only the user's correction caught the stale premise.
• A question's premises about the live system — which domain serves what, what sits behind a proxy, what a live response carries — are probed live before the question reaches the user: a browser, curl, DNS. Production configuration overrides every default in the repo, so a code-derived claim about production is a guess, and "not determinable from the repo" is a prompt to probe, not a finding — one question shipped with both domains' roles wrong from a code-default inference, and the user had to say "open them in a browser and see for yourself".
• Rank a recommendation the way a design is ranked: an option that dissolves the trade-off — removes the tension instead of choosing a side — outranks every option that merely prices it, and the status quo earns the recommendation only when no presented option dissolves. Recommending keep-current-and-document while the option table itself held a design that removed the coarseness handed the user a judgment the reasoning had already settled, and they overruled it as the wrong abstraction.
• When audit lenses split on the same finding, the finding is usually several sub-claims bundled into one sentence, some true and some false — which is exactly why one lens confirms it and another refutes it. Read every lens's reasoning and rule on each sub-claim separately. Resolving the split by picking a winning lens ships whichever half that lens got wrong.

Browser-agent hygiene

• Every agent that drives a browser ends with its browser session closed, and the orchestrator verifies it — a successful close, or an agent's report that it closed its session, is not evidence that the processes died; only a process listing is. After each browser agent completes, and again between waves, sweep to zero with the shipped command:

  bash .claude/scripts/browser-sweep.sh          # list survivors with their ages; exits 1 if any are alive
  bash .claude/scripts/browser-sweep.sh --kill   # kill each survivor by its exact pid, then re-list to prove it
  

• Judge leakage by process count and age, never by reported RSS — paged-out memory hides an order of magnitude (7.6GB visible while roughly 80GB was actually held).

• Leakage comes from sessions accumulating across sequential waves, not from parallel width. Keep lanes parallel; the invariant is verified teardown, not a concurrency cap.

• Kills are the sweep's job precisely because it kills by exact pid after listing what is about to die. Never reach for a pattern kill (pkill -f and relatives) as cleanup or as verification — a pattern that reads as lane-scoped routinely matches unrelated long-running processes on this machine, and listing the pids is the inspection step a pattern kill skips.

Ledger discipline

• Structure the ledger as a STANDING DIRECTIVES head (settled policy, kept current) plus a chronological log. Reground after compaction from the head AND the tail — a tail-only reground lets settled directives fade. Reground with bounded reads — the head block, then targeted slices — never by reading the whole file: a mature ledger runs hundreds of kilobytes, and one full read consumed enough context to force an immediate re-compaction.
• Record launches with their task/run ids, lanes with their base commits, and verdicts with their evidence.
• When a long effort concludes, externalize the durable record (decisions, divergences, audit results) to a permanent artifact — a GitHub issue or PR body — before the scratchpad is cleaned up.

Recovery after an interruption

Step 0 is always to positively enumerate what is alive (task list, workflow roster) — a missing state file or empty output file usually means still-running, not dead. The harness task list can come back empty after a compaction or a host sleep while the underlying shells still run: ps for the actual processes and check whether their log files are still growing before declaring anything dead. Declaring a live task dead and relaunching into its worktree corrupts both. Then, for tasks with positive evidence of death:

• Audit the worktree first: git status/git log. Clean tree at a known commit → relaunch the charter unedited. Dirty tree → launch a continuation agent on the same worktree whose first step is diffing working tree vs last commit to classify done/partial/untouched, then finishing — never a blind restart.
• A QA agent killed mid-mutation-proof can leave deliberately broken code on disk — check for a live mutation before anything else runs there.
• An agent that is still addressable resumes by message from its last checkpoint, keeping its context, instead of restarting cold.
• Probe returned capacity by attempting real work. Never idle-wait.

Shipping and merging a lane

• A PR opens only after the lane's full pre-PR pipeline has run: builder claims verified, full gates green, the adversarial code-review audit complete, and an orchestrator ruling on every confirmed finding. Small diffs are not exempt — momentum after green gates is exactly when the audit step gets dropped, and on a three-line security fix the almost-skipped audit was the one that surfaced a fourth same-class vulnerability.
• Write every PR title and description for an external reader: someone with zero knowledge of the execution that produced the change must understand what it is and why it matters. No orchestration vocabulary (goals, lanes, charters, sanctioned allowances, finder/verifier counts, workspace skills or tooling-repo names), and no superseded project state — describe what the change IS in the product's own terms, and rewrite the body as the work evolves so it always reads as current truth, not history.
• An umbrella PR's body is a deliverable, not a tracker. While work is landing it may be organized around progress, but the effort is not done — and a goal is not met — until the body is rewritten to read as finished: no tracking framing ("Landed", "updated as areas land", in-flight lists), and no links to the internal PRs that built the branch — sub-PR references are noise to an external reader, who needs the result described, not the delivery.
• When a PR whose title promises no behavior change (test, docs, chore types) actually changes production behavior, the title must say so and the body must lead with a complete list of every behavior change — readers won't expect behavior in that diff. Enumerate the list from the real base-to-tip diff of non-test paths, never from memory of the work.
• A defect fix described only by its new behavior reads as housekeeping. Each behavior-change entry states the problem first — what broke, under what conditions, with what consequence — and then the fix, so its real weight is legible: "visit ingestion is idempotent" hides that a routine SQS redelivery permanently broke a paying customer's ad attribution. Write the consequence in the product's terms, not the schema's.
• A before/after contrast in a PR body ("no longer", "used to", "restored", "remains") is only tellable against the PR's base: if the base itself carries the old behavior, the contrast describes the change; if the branch introduced the surface, the contrast narrates the branch's own development history, which the reader never saw — state current truth only. Check each such phrase against the base, not against memory of how the work went. A refute pass does not catch this class on its own — fact-checkers verify claims as stated and confirm a contrast that is factually true but tells internal history — so make base-tellability its own explicit rubric item in any refute pass over a body.
• A branch that backs an open PR — a lane under review, a shared feature branch — syncs by merging its base in, never by rebase or force-push. Rebase is for branches nothing else references yet.
• A lane's pushed commits are never rewritten — a fix pass lands as a follow-up commit, not --amend + force-push, even before any PR exists. Rewriting another agent's pushed commit trips the harness's security review, and the resulting permission blocks can stall the pipeline at PR creation until the user intervenes; a tidy single-commit history buys nothing a squash-merge doesn't.
• Before rebasing, predict the conflict surface: intersect the lane's touched files with what main gained since the lane's base. Empty intersection → expect a clean rebase; non-empty → you know exactly which files need care.
• Run rebase and post-rebase gates as separate, individually-checked steps — a chained command can swallow a mid-rebase conflict's exit code.
• After rebasing an approved commit, run a patch-identity check: diff the rebased patch against the approved patch (sorted added/removed line sets). A rebase is itself a mutation risk — a failed automated edit can commit conflict markers, and only an identity check catches it.
• When two open PRs both carry Django migrations in product-monolith, the second to merge must renumber its migration after the first lands and update the app's max_migration.txt (django-linear-migrations turns the collision into an explicit conflict), then re-run make lint-migrations and the tests against a freshly migrated database. When two lanes touch the same seam, merge the larger diff first and let the smaller one absorb the rebase.
• A lane that changes user-visible copy, CTAs, or navigation runs the touched repo's full suite locally before pushing — a test asserting retired copy can live outside the lane's own files, so only a full-suite run catches it before CI does.
• When merging a lane that changed product behavior, grep the base for tests that pin the OLD behavior — deliberate defect pins included — and run the touched app's full unit package on the merged tree before pushing the merge. A lane branched before a sibling's pin landed is green on its own tip and red on the union, and scoped e2e re-runs never catch a unit-level pin: one merge shipped red because the base carried an honest pin of exactly the bypass the lane had just fixed.
• The dual holds for e2e: e2e specs pin wire payloads the unit suite never re-asserts, because unit tests share the code's own fakes and schemas. When a wave changes the shape of a dispatched payload or another externally visible contract, grep the repo's e2e specs for pins of the changed shape and run the affected specs on the merged tip before pushing — a merged-tip gate set of lint plus unit suites stays green through all of it. Stale pins stack: a second wrong assertion hides behind the first, so rerun each affected spec until it passes rather than fixing only the first failure.
• A clean automatic merge is not a correct merge. Git unions two same-named classes without a murmur (redefinition, F811) and keeps both sides' claims on the same registry, manifest, or census entry. When merged trees both grew one of those, sweep the merged files for duplicated meaning and re-measure the union's counts from the merged tree — never adopt either side's number.
• A PR's CI runs on the merge of its head with its base. On a long-lived feature branch this cuts both ways: lane PRs into the feature branch never see the trunk, so drift on main — a renamed heading, retired copy, a moved route — surfaces only in the feature→main PR's CI, which tests the merge tree. Whenever main advances meaningfully, and always before declaring the feature branch done, sync it — the main-sync skill owns the protocol.
• A merge into a long-lived feature branch does not reliably produce a CI run on the branch's new tip — GitHub does not always emit the pull_request: synchronize event that triggers one. When a criterion requires CI green on a tip, confirm by SHA that a run exists on that exact commit rather than inferring it from the merge; when none does, either dispatch the workflow on that ref (if the repo's CI declares workflow_dispatch) or prove the tip's tree is identical to a tree a passing run already covered.
• A CI watch (gh pr checks --watch, gh run watch --exit-status) is a completion signal only — its exit code is never the verdict: gh run watch --exit-status exited 0 on genuinely failed runs three separate times in one effort. The only verdict is a fresh direct gh pr checks read (plus a mergeable/mergeStateStatus read) taken after the watch returns.
• When the host kills long-lived foreground watches and background poll loops within minutes (this machine does), a --watch is not a reliable completion signal at all: use a persistent Monitor task that polls the run id and emits one event on any terminal state. The fresh-read rule is unchanged — the monitor only tells you when to go look.
• Arm a watch only after the new run has registered under the pushed SHA: poll gh run list --branch <branch> --json databaseId,headSha for the run id and watch that id. A fixed sleep is not enough under queued runners — a 35s wait once bound a watch to a placeholder check that vanished, reading as a false CI failure.
• On a multi-wave effort against an open PR, every intermediate push gets a watch armed this way, and its verdict is read before the next push builds on the tip. Local gates are not the branch's CI — a different matrix, e2e included — and a tip nobody reads stays red invisibly: one branch ran red for two hours and four more pushes until the user flagged it.
• A "failed" CI job with no step concluding failure/cancelled (check gh api .../jobs) is infrastructure death — rerun it. A job with a real failing step is investigated before anything merges. A job hung far past its normal duration is the same class: cancel it and fetch the cancelled attempt's logs (gh api .../actions/runs/<id>/attempts/<n>/logs) to see the exact step it stalled on — a suite-hang diagnosis dissolved into a stuck dependency-install step once the log showed the suite never booted. Recover with gh run rerun <id> --failed, which keeps the run's completed green jobs. When the stall is on an environment step (a dependency download, a browser install), check whether unrelated runs repo-wide hang at the identical step: that signature means external infrastructure, and a rerun armed before the outside service recovers just hangs again — confirm recovery (any fresh run passing that step) before re-arming, and never bypass the gate to merge around it.
• When GitHub itself is unstable, a gh command that reports failure may still have succeeded server-side: after a failed gh pr create, read gh pr list --head <branch> before retrying, and verify exactly one PR exists once it goes through. Declaring a CI failure a flake takes positive evidence — the failing specs are disjoint across runs, each one passes elsewhere on the same tree, and the diff cannot reach the failing path — and even then the reruns that evidence buys are capped at two before someone investigates the failure for real.
• Prepared-ahead scripts (placeholders patched at launch) keep the pipeline saturated, but audit a prepared script's content immediately before launching it — staleness there has shipped wrong charters.
• Fixes to a reviewed feature branch land inside that branch — internal PRs merged into it, exactly as on a goal's feature branch — or are explicitly dropped. Never propose fast-follow PRs to land after the branch merges: everything ships in the branch under review or not at all.