Orchestration

Orchestration

Everything here documents the repository as it is on main. If main disagrees with this file, main wins: follow it and flag the drift.

Delegation only works when the orchestrator treats every subagent claim as unverified and every judgment call as its own. These rules were each paid for by a real incident.

Charters

A charter is the task prompt a subagent receives. Requirements:

• Reference every document by absolute path — a relative path resolves differently (or not at all) in the agent's working directory.
• When parallel lanes could touch the same module, name the sibling lanes and give each an explicit do-not-touch list of files/functions the others own. For genuinely shared append-only files (the MCP tool registry, the jobs array, migrations), say up front that rebase conflicts there resolve keep-both.
• Assign each subagent an explicit scratch-file namespace — its lane or task name as a filename prefix — and forbid it from writing any shared scratchpad file, above all the orchestrator's LEDGER.md. Parallel agents that share a scratch filename clobber each other's state, and a fix agent that writes its own notes over the session ledger destroys the orchestrator's recovery record.
• Include a stop-on-contradiction clause: if the charter's factual premise (a claimed bug, a claimed missing feature, a spec citation) doesn't survive verification against the primary source, the agent halts that item and reports with citations instead of inventing. "Verified already correct, no change made" is a valid, expected outcome. Charter-supplied names — tool names, slugs, feature labels — fall under the same clause: an agent that discovers the real feature differs from the charter's name must correct the name and flag it, not ship correct content under a stale identifier.
• When two reference sources disagree, the charter names the conflict for adjudication — it never silently picks one. A charter's factual claims are no more trustworthy than a builder's until verified.
• A charter for any agent whose output ships anywhere public — code, copy, test fixtures, an issue, a PR body, review comments — forbids personal data from real Discord data (usernames, user ids, message content) unless strictly necessary, and when the source material the charter hands over (an issue, a transcript) itself carries real names, the charter lists those exact strings as a deny-list. Before shipping, the orchestrator greps the diff and the published artifacts for them: the generic prohibition alone has failed twice — a real person's email shipped inside a published investigation issue, and a real name traveled from an issue's text into a tool's schema copy and test fixtures despite a charter clause forbidding it, costing a remediation cycle.
• Instruct agents to run slow verification commands (pnpm run tsc, pnpm run test:unit, pnpm run test:e2e) synchronously in the foreground. A workflow agent that backgrounds a check and ends its turn waiting on a notification is terminally stalled — its turn ending is final.
• Write each new charter/script fresh with the Write tool. Deriving one by programmatically splicing a previous one introduces silent syntax errors.
• A parameterized script must parse args defensively (string-or-object) and hard-throw on any missing required field — a silently-undefined interpolation produces a plausible-looking empty result that reads as a clean pass.
• A slice that ships an MCP tool, a gateway handler, or an ingestion path must drive that surface live at least once — a real MCP client session over stdio, or a scripted gateway event through the real business function — before handoff. A unit-only exit bar passes code whose query crashes at runtime or whose tool schema rejects every real input.
• A live proof binds only the commits that preceded it. Sequence the end-to-end MCP session as the charter's last step, after the final commit, and redo it after any later change to the proven surface — a proof that predates a schema or handler commit is evidence of nothing, and a redone proof regularly catches what the suite missed (a coercion regression, a serialization mismatch).
• A review-fan-out charter carries the orchestrator's accepted-decisions list: every settled divergence and deliberate design ruling the work under review already argued, stated as do-not-report items. Without it, finders re-litigate settled calls and the verify stage burns its budget refuting them; with it, every surviving finding is news. The list names decisions, never files — a finder must still probe the implementation of an accepted decision for holes (an accepted "guard exists" ruling does not shield a hole in that guard).
• A skeptic or verifier charter carries TWO acceptance rubrics, not one: a finding stands if either (a) its concrete-input failure scenario occurs at runtime, or (b) it identifies a violation of a codified house rule or doctrine in the work under review. A runtime-only rubric refutes correct doctrine findings — a migration that models a reversible toggle as one action-enum table where the database-design skill prescribes paired event tables — exactly when the fix is cheapest. Doctrine findings against unmerged schema or shape default to fix-now, because schema shapes are effectively permanent once merged.

Verify, never trust

• Re-run the gates (pnpm run lint, pnpm run tsc, pnpm run test:unit, pnpm run test:e2e) personally before advancing any stage on a subagent's self-report.
• Reconcile numbers exactly: baseline + computed delta = measured, and identical across two runs. A mismatch is a hard stop, not noise.
• A task counts as launched only when its task/run id from the tool result is in the ledger. A written script or a narrated intent is not a running task.
• Independent lanes launch simultaneously, never in staggered waves. Host contention slows every lane's gates but never justifies holding a lane back — an idle lane costs more than a slow gate.
• A review or QA returning zero findings is only a clean pass if its agents all completed. Errored finders, an implausibly fast run, or a wrong agent count mean the run is broken — relaunch it.
• Findings about uncommitted working-tree state from a parallel review fan-out are suspect: when finders share one working tree, one finder's in-flight mutation-proof — code deliberately broken to validate a test, then restored — is visible to its siblings and reports as a phantom defect. Verify any working-tree observation against the commit under review (git show HEAD:file), and either require finders to mutation-prove only against committed state and restore immediately, or discount working-tree-only observations at the verify stage.
• Independently probe a shipping agent's claimed git state (git merge-base <lane> origin/main vs origin/main's tip) before merging. Verify mechanical batch edits landed by grepping the expected before/after state, not by exit code.
• Personally source-read any confirmed "inconsistent with spec/reference" finding before directing the fix — especially when the fix would reverse a prior ruling. A finding's observation can be accurate while its verdict of "wrong" is not.
• An investigation chartered around a stated finding tends to confirm and amplify it: the charter's frame decides what the agent looks for, so a mechanism living in an adjacent flow stays invisible even when the agent's own appendix names it. Before designing on, escalating, or alarming the user with an investigation's headline conclusion, commission one independent second perspective under a different frame (compare against a reference implementation, or charter a refuter) and adjudicate any disagreement personally against primary sources.
• An agent's claim that a symbol has a single consumer ("only the seed imports this") is a factual claim — grep the whole repo, tests/ included, before directing a refactor that removes or narrows a shared contract on its strength. It bites hardest when the claim justifies reversing another agent's deliberate earlier design, and green CI is no safety net when the broken consumer only runs against state CI never exercises — the dev seed against a real configured store is exactly such a consumer.
• Audit PR-body citations ("per spec X", "per source Y at file:line") against the actual cited location, and check any "deferred as polish / out of scope" claim against the actual contract. Builders produce plausible but fabricated attributions.
• Any change to app/framework/ made by a fix/QA agent gets the orchestrator's personal diff read before shipping — its blast radius exceeds any lane's review scope.
• When a builder reports it verified a tool by simulating the call — invoking the business function directly, or hand-constructing a JSON-RPC payload — instead of driving the real MCP server over stdio, treat that as an unverified claim and dispatch a real-session prober before relying on it. A "harness quirk" explanation for why the real client failed is itself a claim to verify.
• Tee gate runs to a scratch log and grep the log, never the live stream — a filter on a live run discards the one failure name needed when a flake appears once and never reproduces.

The release audit

The Definition of Done's code-review audit runs as a fan-out of independent read-only auditors with distinct lenses — doctrine correctness, security and boundaries, product-surface truthfulness — over the merged mainline, after the last build lane lands. Green gates and clean lane reports never substitute for it: every lane verifies its own slice against its own charter, so claims that span lanes (copy asserting what another lane's code can observe, docs describing a mechanism that shipped differently, a README a newcomer walks end to end) go unchecked until something reads the whole. Auditors report; they never fix. Triage every finding into an explicit per-finding ruling — fix, accept with the reason, or defer to the user with options — collect the fix rulings into one charter for a single remediation lane, and verify its PR like any other.

• Every demand gets an architecture round before any delegation: zoom out and map the mechanism behind the ask, where else that mechanism lives, and the repercussions of candidate fixes — then design at the level the root cause lives. A defect surfacing in one MCP tool but rooted in a shared derivation gets the derivation fixed, never a tool-local workaround; a batch of demands gets grouped by mechanism, not by surface, before lanes are cut. Zoom in only with full domain over the systemic impact. This rule was paid for: a local patch of a polling defect owned by a shared framework hook would have left the same bug live in fifteen sibling surfaces, and fixing the hook surfaced a second latent defect in it.
• Design and adjudication are orchestrator-tier work. When the evidence under a design changes (stale reference, corrected spec), redo the design at the orchestrator tier — never hand the stale design down with a "re-verify your citations" instruction.
• Grounding priority when state or policy is uncertain: primary sources (git, live files, the actual spec) beat live skills, and live skills beat ledger notes or compacted memory. On a policy detail, the live skill file always wins over ledger shorthand.
• When parallel lanes independently invent shapes for the same shared surface, adjudicate one canon and give later-shipping lanes an explicit adopt-the-canon duty at rebase.
• An invariant that spans several lanes' data — a shared fixture, a derivation every lane's events must preserve — is orchestrator-tier design work. Derive it personally from primary sources, commit it as a fixture, and hand lanes only consumption rules; never let each lane implement its local view of the global rule. A lane cannot see the whole — two successive lane agents once each handled their own slice of a shared derived balance plausibly and broke the global invariant both times.
• A lane reports out-of-scope findings with repro evidence; it never fixes them inline. The orchestrator's disposition of a reported finding is fix-now by default: a finding that is small, well-scoped, and needs no user ruling gets fixed immediately — folded into the reporting lane's open PR when it owns the file, otherwise a fresh fix lane — never parked. A GitHub issue is legitimate only for a decision the user must make, work blocked on other work, or an unknown-cause investigation someone will actually run. Never file an issue as idea storage — no "maybe someday" refactors, no future-abstraction notes (Shape Up's no-backlogs rule: really important ideas come back on their own; a standing pile only makes everyone feel behind).
• A designer's "too invasive / out of scope" blast-radius claim is a factual claim, not a judgment call — grep it before accepting the compromise it justifies.

Ledger discipline

• Structure the ledger as a STANDING DIRECTIVES head (settled policy, kept current) plus a chronological log. Reground after compaction from the head AND the tail — a tail-only reground lets settled directives fade.
• Record launches with their task/run ids, lanes with their base commits, and verdicts with their evidence.
• When a long effort concludes, externalize the durable record (decisions, divergences, audit results) to a permanent artifact — a GitHub issue or PR body — before the scratchpad is cleaned up.

Recovery after an interruption

Step 0 is always to positively enumerate what is alive (task list, workflow roster) — a missing state file or empty output file usually means still-running, not dead. Declaring a live task dead and relaunching into its working tree corrupts both. Then, for tasks with positive evidence of death:

• Audit the working tree first: git status/git log. Clean tree at a known commit → relaunch the charter unedited. Dirty tree → launch a continuation agent on the same tree whose first step is diffing working tree vs last commit to classify done/partial/untouched, then finishing — never a blind restart.
• A QA agent killed mid-mutation-proof can leave deliberately broken code on disk — check for a live mutation before anything else runs there.
• An agent that is still addressable resumes by message from its last checkpoint, keeping its context, instead of restarting cold — but NEVER message an agent a live workflow still owns. The message forks the transcript into two live instances sharing one working tree: each sees the other's edits as a foreign actor's, and one fork's honorable stop can strand half the work. Deliver mid-flight course corrections to a workflow-owned agent only through the workflow (edit the script and resume the run), or wait for the run to end and relaunch with an amended charter.
• Probe returned capacity by attempting real work. Never idle-wait.

Shipping and merging a lane

• Write every PR title and description for an external reader: someone with zero knowledge of the execution that produced the change must understand what it is and why it matters. No orchestration vocabulary (goals, lanes, charters, finder/verifier counts, workspace skills or tooling-repo names), and no superseded project state — describe what the change IS in the product's own terms, and rewrite the body as the work evolves so it always reads as current truth, not history. This repo is public: every PR body, commit message, and issue is read by strangers.
• When the user shares another organization's work (a PR, a repo, a pattern) as the model for a change here, that source never appears in anything visible in this repo — PR bodies, commit messages, code, comments, or issues. Describe the change entirely on its own terms; the reference lives only in the conversation.
• Merge PRs with a merge commit (gh pr merge --merge). Never squash or rebase-merge. Issue the merge as its own command with nothing chained to it: a merge chained to local git work (gh pr merge … && git fetch && git merge --ff-only) has been refused by the permission classifier where the bare merge passed. Fast-forward the main checkout in the next command.
• Branch protection refuses to merge a branch that is behind main ("the head branch is not up to date with the base branch"), so every merge after the first in a train needs the update ritual: gh pr update-branch <n>, wait for the fresh check run on the new head to complete, one fresh gh pr checks read, then merge. GitHub deletes the remote branch on merge — clean up only the local branch and its worktree, and git remote prune origin for the stale tracking ref.
• Before rebasing, predict the conflict surface: intersect the lane's touched files with what main gained since the lane's base. Empty intersection → expect a clean rebase; non-empty → the files needing care are known exactly.
• On a non-empty intersection, a conflict-free rebase is not evidence of correctness. Git auto-merges a lane's hunks around a sibling's rewrite of the same function, leaving code that references symbols the lane deleted — and the lane's analysis, written against pre-merge main, can assert facts the sibling has since made false. Re-derive the lane's claims about the intersecting files against post-merge main, and resolve the rebase semantically — take the sibling's merged shape as authoritative and re-apply the lane's intent on top — never by accepting whatever auto-merge produced.
• A shared shrink-only registry (a pending list every lane deletes entries from) is resolved by construction, never by conflict-hunk surgery: take the merge base's current copy, delete exactly this lane's originally-removed entries (captured from its diff before the rebase), then verify the removed set stays inside the lane's slice and the count arithmetic closes. Deleting conflict blocks textually silently drops neighboring keep-lines the moment a hunk mixes entries being removed with entries that must stay.
• Run rebase and post-rebase gates as separate, individually-checked steps — a chained command can swallow a mid-rebase conflict's exit code.
• After rebasing an approved commit, run a patch-identity check: diff the rebased patch against the approved patch (sorted added/removed line sets). A rebase is itself a mutation risk — a failed automated edit can commit conflict markers, and only an identity check catches it.
• When two open PRs both carry migrations, the second to merge must rename its migration file to a timestamp later than every migration already on main, then re-rebase and regenerate types (pnpm run db:generate) after the first lands — a migration that is new but timestamped earlier than an already-applied one can break the migrator, and git can auto-merge a semantically wrong types.d.ts. When two lanes touch the same seam, merge the larger diff first and let the smaller one absorb the rebase.
• A lane that changes user-visible copy — tool names, tool descriptions, mapped status copy, README steps — runs the full end-to-end suite (pnpm run test:e2e) before pushing. The unit gate never executes those specs, and a spec asserting retired copy lives outside the lane's own files, so only a full-suite run catches it before CI does.
• A lane that changes product semantics — what a tool returns, what a status means, which reactions count — additionally greps the whole repo for prose stating the old behavior: tool descriptions, error copy, .common.ts copy maps, the README, .env.example comments. Stale copy contradicting shipped semantics survives every gate — no spec asserts it, and it reads fine to a reviewer who doesn't know the new rules — so only the grep catches it before a user does.
• Watch CI with gh pr checks <n> --watch (exit code as verdict, not parsed text). Before merging, do one fresh direct gh pr checks read — never merge on a monitor's word; monitors die on network timeouts and carry parsing bugs.
• A PR showing zero checks is not "CI hasn't started" — after a couple of minutes it means CI will never start. The pull_request workflow runs on the merge commit, so a PR whose branch conflicts with base (gh pr view <n> --json mergeStateStatus → DIRTY) has no merge commit to build and fires no checks at all. The fix is a rebase resolving the conflict.
• Arm gh pr checks <n> --watch only after the push's new check run has registered (sleep ~30s or poll first). A watch armed immediately binds to the superseded run and exits nonzero the moment that run disappears, reading as a false CI failure. A single update-branch can also spawn two check sets moments apart, so a pending-poll can bind the watch to the first while the second becomes the one that counts — the watch then exits green early. When the fresh pre-merge read shows pending checks after a green watch, that is what happened: re-arm the watch on the current set and keep the merge gated on the fresh read.
• Take full commit SHAs only from git rev-parse or the API — never expand a short SHA by guessing. A fabricated SHA in a check-run poll loop waits forever on a ref that does not exist, and its silence reads as CI still pending.
• update-branch mutates the remote PR head with a merge commit, and calling it immediately after a push can 422 on a stale-head race (retry after settling). Fetch and reconcile the remote head before pushing any further commits, or the push is rejected as non-fast-forward.
• A lane's local branch and its PR head branch can differ. Push follow-up commits with git push origin HEAD:<pr-head-branch>, taking the head name from gh pr view --json headRefName, never from the local branch name.
• A "failed" CI job with no step concluding failure/cancelled (check gh api .../jobs) is infrastructure death — rerun it. A job with a real failing step is investigated before anything merges.
• Prepared-ahead scripts (placeholders patched at launch) keep the pipeline saturated, but audit a prepared script's content immediately before launching it — staleness there has shipped wrong charters.