Adversarial Loop

Adversarial loop

Two agents with asymmetric roles and separate contexts, tests before fixes — a second vendor is one way to get that separation, not a requirement (see the tool-posture section). The building agent never grades its own work, and the adversary never merges its own opinion — the merge condition is a green gate plus an exact verdict token.

This exists because diff review after the code is written cannot catch the worst class: a suite that passes without asserting anything. A reviewer reading a diff sees a plausible test. Only a test that was watched failing on the defect is evidence. So the adversary's first deliverable is failing tests, not comments.

Roles

• Adversary (a different vendor's model): audits the subject, writes acceptance tests that FAIL on current code, and reviews each fix round. It never edits the fix branch and never runs the gate.
• Builder (this session): confirms each finding against the real code before starting, implements, runs the complete gate, and owns the evidence. It never edits the test branch.
• Two branches: <topic>/test-acceptance (adversary's commits) and <topic>/fix (builder's commits, which merges the test branch in).

Route by what each side is actually good at

[measured 2026-08-31] over one 24-round audit of a repo's own gates. The adversary's eight opening findings were all real, all in gates the builder had written, and all in code that an 85-suite gate passed green. Every one was the same class: a check that reports success while proving nothing.

So the split is not "a second opinion". It is:

The builder ships past contract defects precisely because its own tests encode its own assumptions. That is the gap the adversary fills, and it is narrow: the adversary cannot run your gates, cannot see your tree, and will keep producing ever-narrower findings well past the point of value.

The adversary needs TOOLS, or you are buying assertions

[measured 2026-08-31] The routing table above was first written crediting the vendor for those eight findings. A controlled test the next day does not support that reading, and the correction is the more useful half.

Same prompt, same unreviewed 214-line authorization gate, neither model having seen it. The cross-vendor adversary returned 11 findings; an in-house subagent on the builder's own model returned 10; about 8 were the same defects. The in-house one additionally caught a live false pass the other missed, and prefixed every finding with "Measured" — it had actually executed the predicate and run the target's selftest against an empty root. The cross-vendor reviewer reasoned statically and asserted.

The confound was the operator's: the subagent was given Bash and Read; the cross-vendor reviewer was given a read-only sandbox. One could verify empirically and did. That is tool posture, not vendor judgment, and it plausibly explains the entire apparent gap.

So the likely source of the original eight findings is this skill's PROTOCOL rather than the second vendor: an adversary required to write tests that FAIL on the defect produces empirical evidence by construction.

Three rules follow, and they outrank the routing table above:

• Grant the adversary enough tools to prove a finding — write access to a scratch area, and the ability to run the thing. An adversary that can only read will hand you plausible reasoning, and plausible reasoning is exactly what a false-verdict defect survives.
• Require demonstration, not assertion. "Measured: printed " in every finding. A finding without a reproduction is a hypothesis, and it costs a builder round to discover which.
• A cross-vendor adversary is optional; the posture is not. Run this loop with an in-house subagent on a separate context first. Reach for a second vendor when you want independence from your own model's blind spots, not because you expect it to see more.

And when comparing two reviewers: an A/B is void unless their tool grants match. Check that before believing any comparison, including the one above.

Reach the adversary over MCP, not a desktop app

[measured 2026-08-31] The same audit was driven by computer-use into a desktop app, and the transport — not the model — produced most of the waste: roughly fifteen click batches lost to focus changes, two stale clipboard re-pastes that burned two entire review cycles, and several stalls waiting for a human to unlock the machine.

An MCP server removes that failure class outright. Measured against the same vendor's CLI on identical prompts:

Two operational notes that cost real time to learn:

• The CLI appends piped stdin to the prompt. Spawning it with an open stdin pipe blocks forever waiting for EOF. Close stdin explicitly.
• Put the commit SHA in every message. It is what catches a duplicate or stale send immediately, instead of spending a review round on already-reviewed work.

Pick the reviewing model deliberately and verify what RAN, not what was asked: a per-call model override is honoured, but read it back from the vendor's own session log before trusting it.

One thread for the whole loop, not one per round

[measured 2026-08-31] Every fresh call to the reviewer re-pays its entire preamble: 22,800 input tokens over MCP, 29,343 over the CLI, before it has read a single line of your diff. A 24-round audit run as 24 fresh calls pays that 24 times and gives the reviewer amnesia between rounds.

The reviewer keeps context only when you reply INTO its thread:

round 1   mcp__codex__codex        { prompt: <audit brief>, cwd, sandbox, model }
          -> { threadId, content }          <- record this threadId
round 2+  mcp__codex__codex-reply  { threadId, prompt: <round N brief> }

Verified: a reply into an existing thread recalled the reviewer's own previous answer. A fresh codex call returns a NEW threadId every time and remembers nothing, and the CLI's exec is fresh per invocation as well.

What this changes about how a round reads. With a live thread you no longer restate what the reviewer already knows, so a round brief shrinks to the delta:

• the commit range for THIS round, <prev-tip>..<new-tip>
• what you changed per blocker, one clause each
• the gate result, with the command that produced it
• the exact verdict token you want back

Keep the SHA in every message even so. The thread gives the reviewer memory; it does not tell you whether your send landed, and a duplicate send is the failure that costs a whole round.

Record the threadId in the round log (the row shape is defined once, under "Measure every run"). It is the only handle that lets a later session resume the same review, and it is cheap to lose.

When to start a NEW thread instead. A thread carries the reviewer's earlier conclusions, which is the point, and also its earlier mistakes. Start fresh when the subject changes, when the acceptance contract is re-frozen, or when you deliberately want a second opinion uncontaminated by round 1. An independent second read is worth more from a new thread than from the one that already agreed with you.

The loop

1. Audit round. The adversary reads the subject and returns numbered findings (F1, F2, …). The builder confirms or refutes each against the real code before any work starts — an audit finding is a claim, not a work item.
2. Tests first. For each confirmed finding the adversary commits a test that fails on current code. The builder runs each new test and checks it fails for the stated reason — a test failing on a typo validates nothing. A test that passes on current code goes back: it does not encode the defect.
3. Fix. The builder implements on the fix branch until the acceptance tests pass, then runs the repo's complete gate.
4. Review round. The adversary gets the commit range (git diff <test-tip>..<fix-tip>) as a REPLY into the loop's existing thread, not a fresh call, and answers with either new blockers or the exact token VERDICT: CLEAN. Nothing else counts as approval — prose verdicts get misread, so the token is agreed up front and matched exactly.
5. Loop or land. Blockers go back to step 2 or 3. On the token AND a green gate, merge. Neither alone is sufficient.

Bounding the rounds, and why a round cap is not the bound

The obvious control is a round cap. It does not hold: the first production run of this loop was capped at 5 and ran to 24, because every round raised a real blocker in the previous round's fix. Nobody was padding. The adversary was asked whether it had softened under pressure and answered that it had rejected "clean" through four consecutive rounds and reached it on a full outcome truth table. A cap you would be wrong to enforce is not a bound.

What actually converges the loop is narrowing what each round may reopen:

• Freeze the acceptance contract after the audit round. The tests define done. A blocker that does not map to a frozen test is a new audit, and it waits for the next loop rather than extending this one.
• Watch for the round that stops being about findings. Rounds 1–11 of that run fixed the subject and produced a design change that deleted 151 lines of guard code. Rounds 20–24 were five rounds around one exit-classification decision. Same rigour, a fraction of the value — that transition is the signal to freeze, not the round number.
• Run the targeted check during iteration and the complete gate at convergence. Fourteen full 85-suite gate runs were safe and mostly wasted; the affected suite answers a micro-fix in seconds.
• A human reads the delta every round — raised, fixed, refuted — and decides whether another round buys anything. That judgement is cheap outside the loop and re-bills a full context inside it.

Make the decay observable: a round log

"Watch for the round that stops being about findings" is a judgement, and a judgement made inside a long loop is made by the party least able to make it. Record it instead.

Use the round log the "Measure every run" section already defines. The row shape is written down exactly once, there, and it carries the disposition columns this rule needs. Do not restate it here and do not start a second log: two schemas means one of them is wrong and nothing reports which.

That was worth learning the hard way. An earlier attempt to fix a two-schema contradiction restated the row "with one column added" — which was actually two columns added and two removed, producing a third schema while claiming to extend the first. A schema repeated is a schema that will diverge. Reference it; never copy it.

Findings-per-round is the wrong metric and repeat-subject is not the fix. The naive count points the wrong way: rounds 20 to 24 of the run above raised five real findings, so the rate looked healthy while the loop circled one exit-classification decision.

An earlier version of this section said to count distinct subjects and stop after two consecutive rounds on already-seen ones. The worked example above falsifies that rule — those five rounds sit on one subject and all five were real, so the rule stops at round 21 and loses three acknowledged defects. "Subject" also has no stable granularity, so a caller can make the same sequence converge or not by calling it one subsystem or five clauses.

What decays is the DISPOSITION of what arrives. Give every finding a stable id and classify it as exactly one of:

Only the last two are churn. A round of regression in fix is the loop working exactly as intended, however familiar the subject looks.

This decides when to stop ATTACKING. It does not decide when to merge. The merge invariant is unchanged and stated three times elsewhere in this file: the loop ends on the exact verdict token, and neither the token nor a green gate is sufficient alone. A decayed yield means stop spending rounds and go get that verdict on what is fixed — never land without it.

The stop rule, computed by the caller from the log:

• Two consecutive rounds producing only same unresolved decision and new scope means the loop has converged on argument rather than defects. Freeze the contract, then close the loop the normal way. A single new defect or regression in fix in either round resets this.
• The same id disposed same unresolved decision three times is a decision, not a defect. Take it out of the loop and decide it directly; it will not converge by being attacked again.
• A repeated subject is a prompt to look, never evidence on its own. It tells you to read the dispositions; it does not tell you the loop is done.
• New scope is a separate loop, always. A finding that does not map to a frozen acceptance test is the next audit's input. Logging it in the new scope column is how it survives without extending this run.

The stop rule that matters most: a rising wrongly fixed count

Every rule above answers "has the loop finished?". This one answers a different and more expensive question: "is the loop making things worse?" — and it is the only stop condition that fires while findings are still real.

[measured 2026-09-01] A five-round run on a static analysis check. The wrongly fixed column, meaning a repair that made something worse than it was:

fixed plateaus and wrongly fixed climbs. Every round the builder repaired real defects and introduced new ones at a similar rate, so no round was worthless and no round converged. The adversary was not padding: each finding came with a runnable counterexample, and the builder independently reproduced the last one before accepting it.

A rising wrongly fixed count means the DESIGN is wrong, not the patches. That run refuted three designs for the same sub-problem — parenthesis balancing, then continuation-on-a-trailing-operator, then a fixed radius — and each replacement was a considered response to the previous refutation. The third was accompanied by the claim that its only failure mode was being too small; a nine-line fixture disproved that in one round. When three honest attempts at one sub-problem all fail, the sub-problem is the finding.

Two things to do instead of round six:

• Ask what the checker would have to KNOW to be right, and whether it can know it. That run needed a JavaScript parser to find where a call ends. No parser was available, so every design was an approximation, and approximating is fine — claiming gate-grade precision from one is not.
• Downgrade rather than keep patching. A check whose false positives are demonstrated must not gate; advisory costs a wrong line of output, gating costs a wrong red build and then gets muted. Landing it advisory keeps the real value and drops the false claim.

Record the run as unconverged and say so. A loop that stops without a clean verdict is an honest outcome with a written reason. A loop that keeps going until someone runs out of patience produces the same code and no reason.

The caller computes this, never the adversary. An adversary asked to declare itself done does not: measured across two workflows, one told to output a dry verdict when the work was sound produced zero dry passes and ran to its agent cap. A model asked to find findings finds findings. The log is read from outside the loop, where the judgement costs one cheap read instead of re-billing a full context inside it.

The log is also the only artifact that survives the run. A loop whose history lives in one session's chat cannot be audited, cannot be resumed by anyone else, and cannot tell a later reader whether round 20 was rigour or churn.

Verify the adversary's tests too

The adversary's tests are gates, and a gate's first run is a measurement. Mutation-test them: stub the subject, confirm the suite goes red; restore, confirm green. In the first production run of this loop, mutation testing found two defects in the adversary's own tests (a canary that emitted no usable coverage, and a prepend that broke the target's shebang). The adversary fixed its own tests when shown the evidence — that exchange is the loop working, not the loop failing.

Two sweep rules that bite here:

• A mutation sweep needs the repo to itself. Never run the gate, a diff, or a restore while one is in flight — a killed sweep leaves stub prepends in suite files that read exactly like someone else's broken commit. Check running processes before blaming the diff.
• Restoration is part of the gate. A sweep that dies mid-run must leave the tree recoverable (git status clean after its own recovery), and its exit code must distinguish clean / findings / could-not-restore.

Cross-vendor handover limits

• Structured payloads truncate. One harness cut every structured reply at a hard 2048-character cap, severed mid-token, five out of five attempts — each retry burning a full generation. Keep schema string fields short and push detail into repo files; the repo is the channel that does not truncate.
• Before re-running a failed expensive agent, read its transcript for rejected tool inputs. A payload pinned at exactly the cap means the work is finished and only the handover failed; recover it instead of paying for the run twice.
• The repo is the only shared memory. Findings and test contracts live in committed files, never in one session's chat. The round log is the exception and lives in the gitignored reports directory; it is durable by being appended every round rather than by being committed. The other vendor cannot read your context window.

A timeout aborts the call, not the process

A timeout ends your handle on the adversary. It may or may not end the adversary, and you cannot tell which from the error.

Both outcomes are observed. One dispatch kept working for about ten minutes after the caller had recorded it as failed, rewriting its deliverable four times. A second, on the same machine the same day, died with its transport: its session log froze at the moment of injection and never grew. Treat "still running" as the case to rule out, not as the rule — the cost is asymmetric, because acting on a dead delegate wastes a re-dispatch while acting on a live one corrupts a file.

So the first move after a timeout is a liveness check, not a decision. Look for the process, and check whether the delegate's session log is still growing. Log SIZE is not the signal: a log can be large purely from the context injected at start, which is what fooled the caller in the second case.

The rest follows once you know which case you are in:

• Never re-dispatch before the liveness check answers. The two branches are different work, and the section above exists to tell them apart:
  • Still running → do not re-dispatch, wait. A second dispatch puts two writers on one deliverable, and that is not a race you lose loudly: one file vanished mid-rewrite and was later found 415 lines shorter under an append that looked clean.
  • Dead → re-dispatch, and narrow it first. Nothing is writing, so the only cost of waiting is the whole run. Re-send with whatever stalled it removed rather than repeating the brief that died.
• An empty result is a claim about the handover, never about the work. That holds on both branches, which is why the liveness check comes before any conclusion about what the delegate produced.
• The tell is git show --numstat, not the tool output. Nothing in a tool result announces that a file got smaller. A deliverable that shrank and a deliverable that was edited look identical until you count lines.
• Give the deliverable a recovery path, and pick the one its location allows. An untracked file has none, so an overwrite is total loss rather than a diff. If the deliverable is repo content, commit it to the working branch as it grows — that is a local commit, and pushing still needs the usual say-so. If it is an audit report, it belongs under .claude/reports/, which this repo gitignores deliberately so raw audit output cannot be staged; there the recovery path is appending every round plus the delegate's own session log, not a commit. Do not relocate a report in order to commit it.
• Last resort: the adversary's own session log on disk holds the bytes verbatim. That is what recovery came from when the untracked file was gone. It is a backstop, not a plan — committing is the plan.

Recovery is a stitching job, because incremental writing fragments it

[measured 2026-09-01] The two halves of this section pull against each other, and it is worth knowing before you need it. Writing incrementally is what stops a timeout losing everything. It is also why no single payload in the log holds the finished file.

A 32,327-byte report was destroyed by a git worktree remove — the reports directory is gitignored, so the deliverable was untracked and the removal took it. Recovering it from the delegate's session log:

So a recovery script cannot take the last write or the biggest one. It has to collect every payload, split on headings, and keep the longest copy of each — a later append can extend a section an earlier one truncated, so longest-wins rather than last-wins. Byte count will not match the original, because the log holds what was written and not the file's final assembly; count the SECTIONS recovered against the sections you expect.

Two guards that made the result trustworthy rather than plausible:

• Run a known-positive before believing a gap. The first attempt reported the report missing from the log entirely. The control failed too, which meant the probe was pointed at the wrong session — --last and newest-mtime both pick the wrong file when several runs overlap. Address the session by id.
• Count against the delegate's own closing message. It said 10 HIGH and 1 MEDIUM; recovery produced exactly 11 sections. A grep for headings had said 12, and the grep was wrong. Reconcile against something with a different provenance before declaring anything lost.

This is not licence to move the report. An audit report belongs under .claude/reports/, gitignored, for the reason stated above: raw audit output must not be stageable. The cheap protection is not relocating it — it is remembering that removing a worktree destroys everything ignored inside it, so copy the report out before running git worktree remove, and treat the delegate's session log as the recovery path when you forget.

Instruct the adversary to write incrementally. Append findings to a file as it goes rather than composing one answer at the end, and prefer a format that survives truncation — one finding per block, never a single JSON object, which parses as nothing when cut anywhere. A run that times out then still leaves output, and a partial deliverable beats an empty one every time.

Keep from the dispatch whatever tends to stall it. In the run this was measured on, browser work was the stalling half and was also the half the caller could do directly. Removing it and re-dispatching narrowed produced a complete plan on the next attempt. Split a brief along "what can hang" rather than along what is conceptually tidy.

Measure every run

Append one row per round to .claude/reports/adversarial-loop-<topic>.md in the repo under audit, and total it when the loop closes. This is the only place the row is defined. Anything else that needs it references this line rather than restating it, because a schema written twice diverges.

| round | range | threadId | raised | new defect | regression | unresolved | new scope | tests added | verdict |

The four middle columns are the dispositions from the bounding section, and they replace a single confirmed/refuted pair: knowing a finding was real does not tell you whether the loop is still finding things.

That path is gitignored in this repo, deliberately, so raw audit output cannot be staged. The log is durable because it is appended every round, not because it is committed. Do not describe it as committed and do not relocate it in order to commit it.

Close with: rounds to clean, defects caught after the builder first believed it was done, and defects found in the adversary's own tests. That last number is the honest one — if it is always zero, nobody is mutation-testing the tests, and the loop has degraded into review theatre.

Worked example — the first production run, an 8-finding audit of a plugin repo's own gates, merged as one squashed PR:

The builder believed the work was done after round 1. What the following 23 rounds bought was not polish: they replaced three successive restore strategies that each lost a concurrent writer's edit, deleted an entire lock/nonce/announce protocol in favour of isolation, and established that an infrastructure failure must never be scoreable as test evidence. Two of them found defects in the acceptance tests themselves.

The honest cost line beside that: the last five rounds circled a single exit code, and the loop kept its rigour long after it had stopped buying much.