Gate Toolchain

Gate Toolchain: the right instrument, in diff mode

Two gates carry this pack: CRAP over what the coder just wrote (C6), and mutation testing over its tests (C7). Both are language-native. The scorer has to read the coverage artifact your test run already produced, and the mutation runner has to speak your compiler. This island is the lookup table between a language and those two instruments. It also carries the one fact that decides whether a gate can live inside the agent loop at all: does the tool have an incremental or diff mode?

Every row below is transcribed from the pack's two research briefs, crap-metric.md and mutation-testing.md. Where a brief records no mode, the cell says so rather than guessing. An invented flag is worse than an empty cell.

The CRAP map

Three of these are Martin's own repos, still updated. That is also the reason to read them rather than adopt them. Pointing at an exemplar instead of downloading it is essence-pointer's concern (C23), not this island's.

The mutation map

Which mutants a scoped run then selects is mutant-hunt's contract, not this table's: the story diff intersected with covered lines, plus the tool's coverage-targeted mode. The briefs record coverage-targeting as a general acceleration. They name no tool's option for it, so no option name is transcribed here.

That last cell is the whole argument in one line. Google mutates only diff-touched, covered, non-arid lines, one mutant per line, at ~2B LOC (mutation-testing.md). Scale did not buy them whole-repo runs; scoping did.

The hard rule: incremental, or outside the loop

Inside the agent loop, a gate runs in incremental or diff mode. Whole-repo runs are scheduled work, not loop work.

The reason is the margin, and the margin is the accounting unit (C5): "eventually you will slow the agents down to the point where they're slower than humans. And at that point you've lost the game". A mutation pass that takes an hour on every story does not harden the pipeline. It eats the thing the pipeline was built to produce. The same arithmetic revived both gates in the first place: mutation testing that once meant an overnight run now means "maybe it took it 30 minutes instead of an overnight run" (C7), and CRAP that once meant walking every function by hand now means "run crap over everything you've just done" (C6). Thirty minutes is already a stretch for a per-story gate; thirty minutes times every story is a lost game.

So each declared gate takes one of three shapes, and the shape is stated, never assumed:

1. Tool has a native incremental/diff mode (crap4java --changed, PIT incremental, Stryker incremental). Run it in that mode inside the loop.
2. Tool has none, but takes a file or line filter. Feed it the diff's ranges and keep it in the loop. Computing those ranges is mutant-hunt's script, not this island's.
3. Tool has neither. It runs on a schedule outside the loop, and every story it did not cover is unverified until that run lands. Say that out loud in the evidence; a nightly job is not a per-story gate.

Declare the choice where a script can read it

A toolchain picked in conversation decays; a toolchain declared in a file gets checked. Write one TSV row per gate you actually run:

language<TAB>gate<TAB>tool<TAB>scope_arg<TAB>command
java	crap	crap4java	--changed	crap4java --changed --threshold 6
typescript	mutation	stryker	--incremental	./node_modules/.bin/stryker run --incremental

scripts/toolchain-check.py reads that manifest and enforces exactly four claims per row. The tool is the mapped implementation for that language and gate. The command names that tool. The run is really diff-scoped: the declared flag is flag-shaped, present in the command, not negated, and no whole-repo flag sits in either column. And the command reaches its binary locally: a fetch-and-run launcher is rejected, because a row that says npx stryker resolves an unpinned package off a registry and executes whatever answers, which is a supply-chain decision wearing a gate's clothes. The launcher is matched on its basename, so path-qualifying it (/usr/bin/npx) is the same refusal as the bare name, and a long spelling (bun x, uv tool run) is refused with the alias it expands. Declare the path your lockfile pins (./node_modules/.bin/stryker, mvn, cargo), or pin the launcher with --no-install/--offline.

python3 scripts/toolchain-check.py [manifest.tsv]   # stdin when no file
# exit 0 every row mapped and diff-scoped (--help also prints usage and exits 0)
# exit 1 verdict, a row breaches
# exit 2 usage / IO / closed or unflushable stdout / closed stdin / undecodable /
#        malformed or empty / any other internal error (fail closed) · exit 130
#        interrupted (Ctrl-C, converted to the shell's convention 130 by a normal
#        exit — not re-raised, so a waitpid parent sees an ordinary status, never a
#        signal death). 0, 1, 2, 130 — no other code, and that is sealed rather than
#        assumed: the tail catches argparse's own SystemExit and flushes both
#        streams itself, so a dead output pipe cannot swap CPython's shutdown
#        code 120 in over the verdict. Both spellings are probed below.

The scope column is what makes the declaration falsifiable. Before checking it, the script strips shell comments from the command, then splits what survives the way a shell resolves quotes and escapes before any expansion. shlex does that splitting. When shlex refuses the command outright (unbalanced quotes, a trailing lone backslash), a fallback splits on whitespace and resolves the same escapes and quote marks by hand, so broken quoting cannot smuggle a deny-listed flag past the scan. Every token then keys through one normaliser: Unicode NFC, which also drops a U+FEFF byte-order mark.

Stripping and splitting are two passes, not one parser, so the claim is only the one they are checked on: both agree where a word starts, and therefore where a comment begins. The cut honours the same backslash escaping shlex does. So --msg 'it'''s # 1' --whole-repo keeps its trailing flag instead of losing it to a "comment", and a # that does not start a word (-Dmessage=fix#123) is not a comment at all. "--whole-repo", --whole"-"repo and a BOM-prefixed java are the words they are, not new ones.

Both columns are read, because a gate can be switched off in either. Negation wins wherever it sits, and however the scope is spelled. Every token sharing the declared flag's head is scanned, so --incremental --incremental=false and --incremental=true --incremental=false are both rejected in either order: a command that both enables and disables its own scope is not a scoped run. --incremental=false declared is rejected too, a scope switched off being no scope. Negation means an =-valued spelling: false/no/off/none, empty, or a value that reads as zero (=0, =00, =0.0). A space-separated --incremental false is two tokens, and is not read as negation. That limit is fixtured below.

The whole-repo deny-list reads both columns as well, comparing the flag head, so --all=true falls to the same rule as --all. mutmut run --whole-repo breaches even when the scope column declares a clean --incremental, because the deny-listed flag sits in the command that actually runs. Beyond that: --incremental declared beside a bare stryker run is rejected as absent, as is one parked behind a #, and a non-flag word like run is rejected as not flag-shaped.

Neither the scope check nor the tool check can be satisfied by a substring, but they anchor differently. A bare scope declaration matches on the flag head, so --in-diff is answered by --in-diff=story.diff. A declaration carrying its own value must match the whole token, so --in-diff=story.diff is not answered by --in-diff=other.diff. Tool matching runs on whole squashed segments. That is why echo --changed is rejected as naming no tool, and so is echo --changed -Dcapital=1 declaring pit.

Red/green proof. All seven fixtures live beside the script. Recompute from this island's directory; these thirteen commands gave these exit codes:

python3 scripts/toolchain-check.py scripts/fixtures/dirty-whole-repo.tsv      # exit 1 — 17 declarations, 16 breaching
python3 scripts/toolchain-check.py scripts/fixtures/clean-diff-scoped.tsv     # exit 0 — 7 declarations, 0 breaching
python3 scripts/toolchain-check.py scripts/fixtures/bom-prefixed-clean.tsv    # exit 0 — 7 declarations, 0 breaching
python3 scripts/toolchain-check.py scripts/fixtures/limit-space-separated.tsv # exit 0 — the disclosed limit, run
python3 scripts/toolchain-check.py scripts/fixtures/remote-runner.tsv         # exit 1 — 11 declarations, 8 breaching
python3 scripts/toolchain-check.py scripts/fixtures/tab-hidden-row.tsv        # exit 2 — malformed: line 5: empty field
python3 scripts/toolchain-check.py scripts/fixtures/undecodable-bytes.tsv     # exit 2 — input error: 'utf-8' codec can't decode byte 0xff …
python3 scripts/toolchain-check.py 0<&-                                       # exit 2 — input error: stdin is closed
bash -c 'PYTHONIOENCODING=ascii python3 scripts/toolchain-check.py scripts/fixtures/clean-diff-scoped.tsv'  # exit 0 — text degrades, verdict stands
python3 scripts/toolchain-check.py scripts/fixtures/clean-diff-scoped.tsv >&-  # exit 2 — input error: stdout is closed, never a verdict
# the two paths that used to exit 120, plus the interrupt — returncode captured, not piped, and re-raised as the wrapper's own exit so the pack verifier re-runs each row:
python3 -c 'import os,subprocess,sys;r,w=os.pipe();os.close(r);sys.exit(subprocess.run([sys.executable,"scripts/toolchain-check.py","--nope"],stderr=w,stdout=subprocess.DEVNULL).returncode)'    # exit 2 — usage error onto a dead stderr, not 120
python3 -c 'import os,subprocess,sys;r,w=os.pipe();os.close(r);sys.exit(subprocess.run([sys.executable,"scripts/toolchain-check.py","--help"],stdout=w,stderr=subprocess.DEVNULL).returncode)'    # exit 2 — --help onto a dead stdout, not 120
python3 -c 'import signal,subprocess,sys,time;p=subprocess.Popen([sys.executable,"scripts/toolchain-check.py"],stdin=subprocess.PIPE,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL);time.sleep(0.6);p.send_signal(signal.SIGINT);sys.exit(p.wait())'  # exit 130 — the shell convention for SIGINT, by normal exit, never a verdict

The dirty fixture fails for sixteen distinct right reasons, while its first row passes: a --whole-repo scope, its --all=true spelling, crap4java declared as Go's scorer, a --incremental claim absent from the Stryker command, one hidden behind a #, one negated as --incremental=false, one negated in the scope column itself, one negated by =00, one carrying both spellings at once, a valued --incremental=true answered by --incremental=false, a clean --incremental declared over a command that still runs --whole-repo, the same breach in quotes ("--whole-repo"), the same breach hidden behind the ''' escape idiom and a #, the same breach after a mid-word fix#123, an echo --changed that never invokes the tool it declares, and an echo --changed -Dcapital=1 that declares pit and only buries those letters inside capital.

The clean fixture answers in the other direction. It holds a repository-local path binary (./node_modules/.bin/stryker), a two-word binary (cargo mutants), a Maven coordinate (org.pitest:pitest-maven), a legitimately quoted # argument, a valued scope (--incremental=true) honoured by its command, and a non-ASCII path. The gate may reject none of them. So the pair proves the gate discriminates rather than rejecting every file.

Everything that is not a verdict exits 2, never 1 and never 120: malformed rows, a row hidden behind a tab-led # (comment detection is anchored to column 1, so no row vanishes unannounced), a missing file, a directory, closed stdin, an empty manifest, a manifest the decoder chokes on, a write that fails on a closed stdout, and a usage error or a --help whose output pipe is already dead. An input error must never read as a verdict. A latin-1 em-dash pasted from an editor is a bad file, not a bad row. A UTF-8 BOM is the other side of that principle: it is a file-format artifact, so it is stripped and judged as the word it prefixes, never turned into a bad row. Deleting any fixture returns the gate to unverified.

One disclosed limit, fixtured not fixed. limit-space-separated.tsv exits 0 on ./node_modules/.bin/stryker run --incremental false. The scope reads present because negation is only read in its =-valued spellings, and a bare false in the next token is as often a positional argument as a value. Widening the rule would buy this case at the price of false reds on --in-diff 0-shaped arguments, so the claim is narrowed instead: verify a space-separated boolean against your tool once, the way the advisory below already asks you to verify the flag exists.

Boundaries: what this island owns and what it points at

This island owns tool selection per language and each tool's input and mode facts. Nothing else:

• The CRAP formula, its threshold regimes, and the score's coverage hole belong to crap-gate. It decides what number fails; this island decides which binary computes it.
• The mutation gate contract (diff scoping, zero non-equivalent survivors, kill-tasks, the budget cap) belongs to mutant-hunt. It decides what the run must prove; this island decides which runner your language has, and whether that runner can be scoped at all.
• Where the tool executes (pre-commit, PostToolUse, a CI step, a nightly schedule) belongs to agent-guardrails, and the loop machinery around it to archipelago.
• The captured tool report enters evidence-packet format as a verification-ladder rung, never a second evidence format.
• Choosing a model or an agent for a task is model-routing's axis. This island routes nothing but binaries.

Enforced vs advisory

• enforced — the manifest verdict. toolchain-check.py rejects a tool that is not the mapped implementation for its language and gate. It rejects a command whose tokens never name that tool as a run of whole squashed segments. It rejects the named whole-repo flags (the DENY_SCOPE list, matched on the flag head after shell-quoting and backslash escaping are resolved, on the shlex path and the fallback alike) in either column: declared as the scope, or merely sitting in the command that runs. It rejects any non-flag-shaped scope, and a scope argument absent from the command it claims to be part of. It rejects a negated scope in either column, bare or valued: hidden in a shell comment, answered anywhere in the command by an =false/=off/=0-valued twin, present and negated in the same command in either order, or declared falsey itself. It rejects a fetch-and-run launcher (npx, pnpm/yarn dlx, bunx/bun x, uvx/uv tool run, npm exec/npm x, pipx run) in the command, compared on the executable's basename, so path-qualifying the launcher buys nothing: /usr/bin/npx and C:/tools/npx.cmd are the refusal npx already was, and a gate row cannot install an unpinned package by being followed. remote-runner.tsv proves those reds beside three greens — two lockfile-pinned paths, one of them absolute, and a --no-install-pinned launcher. It reports every malformed row rather than dropping any. And it exits 2 fail-closed on missing, unreadable, closed-stdin, undecodable, malformed, or empty input, on a closed or unflushable stdout, and on any other internal error, argparse's own usage exit included, so no error path can wear a verdict's code or CPython's 120. Proven red and green above.
• enforced — island shape: the pack validator (scripts/validate-island.py at the pack root) gates this file's frontmatter, sidecar, ledger citations, script syntax, and line budget.
• advisory — that the flag you declared exists in the tool's CLI, and that the named tool is the process that runs. The gate checks the flag is flag-shaped and really in your command; it cannot know your tool's option set. A declared --diff that the tool ignores passes the gate and fails reality, so verify the flag against the tool once, then let the gate hold it. The tool check likewise compares names with case and separators ignored. It is anchored to whole segments but matched across token joins, so cargo mutants satisfies cargo-mutants and org.pitest:pitest-maven satisfies pitest: it catches a command that never mentions the tool, not one that mentions it in passing (echo mutmut --incremental passes). Whole-repo spellings outside DENY_SCOPE (--full-run) still pass in both columns, because the deny-list is a named set, not a semantic judgement. Negation outside its =-valued spellings passes too, per the fixtured limit above. The launcher rule is a named set on the same terms: a launcher outside it, one reached through a wrapper script or a shell alias of your own, or a shell that resolves one indirectly ($RUNNER stryker), passes. A command field is free text and no token test reads every spelling. Basename comparison closes the path spelling and only that: a Windows path spelled with backslashes is resolved as shell escapes before the comparison, so C:\tools\npx.cmd still passes where its forward-slash twin is refused. It fails closed in the other direction — a space-separated path argument whose own basename is a launcher name (--in-diff build/npx) reads as a launcher, though the attached spelling --in-diff=build/npx does not. And the --no-install/--offline pin is read as present in the command, not as bound to the launcher it should govern.
• advisory — the map rows themselves. They are transcribed from the research briefs at a point in time; tools ship, rename, and gain modes. A row marked not recorded is an instruction to go look, not a verdict that no mode exists.
• advisory — the three shapes above (native mode / filter-fed / scheduled outside the loop) and the choice among them. No mechanical check today measures your gate's wall-clock against the margin; margin-ledger is where that accounting lives.

Done means

• Every gate you run appears as a manifest row: language, gate, tool, scope flag, real command
• toolchain-check.py exits 0 over that manifest
• Every command invokes the binary your lockfile pins, not a launcher that would fetch one
• Each declared scope flag verified once against its tool's own documentation (the advisory gap above, closed by hand)
• Any tool with no incremental, diff, or filter mode is stated as scheduled-outside-the-loop, and the stories it has not covered are marked unverified

An open box means the toolchain claim stays unverified. Fix the row (swap the tool, add the scoping flag, or move the run out of the loop), re-run the checker, re-check the boxes.

A gate the loop cannot afford is not a gate, it is a nightly job wearing a gate's name (C5).