# Ground Truth Gates

> Build executable verification gates (golden set, replay corpus, project checks) so "it works" becomes a checked fact instead of a claim. Load when changing any LLM-judgment step (classify/extract/route/prompt), refactoring logic that processes real logged data, designing tests for a fix, setting up a commit/ship gate for a project, designing a runtime guard (a hook, validator, or auth check) and its fail direction, or when you are about to trust a passing test that has never been shown able to fail. Also the reference for what "proof gate" means in delegation-and-review packets. Do NOT load for one-off scripts or exploratory spikes — plain operational-rigor covers those.

- Skill: `f-e-u-e-r/ground-truth-gates` (Agent Skill, multi-file: 9 files)
- Install (CLI): `npx skillmds@latest add f-e-u-e-r/ground-truth-gates`
- Raw SKILL.md: https://api.skillmd.com/api/skills/f-e-u-e-r/ground-truth-gates/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: F-e-u-e-r (https://skillmd.com/u/f-e-u-e-r)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/f-e-u-e-r/ground-truth-gates

---


# Ground-Truth Gates

**The core finding:** more prose rules do not improve a capable model on
verifiable work — its gating habits are already native. What is missing is
**something to gate against**. Invest in executable ground truth, not in
longer instructions. Build gates first where judgment work happens
(classification, extraction, routing, prompt output) — that is where habits
are weakest and where a gate converts open-ended quality into a number plus
a diff.

## The one command

Once `template/` has been copied into the project as `checks/` (wire-up below):

```bash
bash checks/run-all.sh
```

Discovers every `checks/*/run.mjs` (plus optional `checks/project.sh`), runs
each, prints `PASS`/`FAIL` per gate, exits non-zero if any fail. That is the
commit/ship gate: "all green" stops being a claim and becomes a checked fact.

## The three gates

| Gate | Question it answers | Where it pays |
|---|---|---|
| **golden** | "Is this prompt/classifier actually better, by how much, and which cases does it miss?" | LLM-judgment steps. |
| **replay** | "Did my change alter output on real logged inputs, and exactly where?" | Refactors and regex/prompt tweaks over production data — catches silent drift reading the code cannot see. |
| **project** | "Do build/tests/types/lint pass?" | Drop a `checks/project.sh` with `npm test`, `tsc --noEmit`, an SCA scan failing on critical/known-exploited (`npm audit` / `pip-audit`), etc. |

A starter implementation lives in this skill's `template/` directory —
copy it into the project as `checks/` and wire it up (~15 min per gate):

**golden:** replace `golden/cases.jsonl` with 30–50 *real, hand-labeled*
examples (`{"input": ..., "label": ...}` per line) — a tiny set is gameable;
a perfect score on a small set is an overfit warning, not a win. Replace
`classify()` in `golden/run.mjs` with a call to the real system (keep it
deterministic per input). Set the team's bar by editing `MIN_DEFAULT` in
`golden/run.mjs` — that is what `run-all.sh` (and any hook/CI on top of it)
enforces; the `--min` flag only overrides ad-hoc runs.

These rules make the golden gate earn its keep:

- **Anonymize structure-preserving** — replace PII values with same-shape
  stand-ins (digits for digits, `client@example.com` for an email, a
  placeholder name like `Jordan Lee` for a name). `REDACTED` destroys the
  very shapes the logic keys on.
- **Include hard negatives** — real inputs that look like a match but must
  fall through. That is where regressions hide and where synthetic cases
  never go.
- **Score cost-asymmetrically** — name the class of wrong output that
  triggers a real, unconfirmed action (wrong route, wrong send) and treat
  any instance of it as a hard failure, not something aggregate accuracy can
  average away. The starter `run.mjs` implements this: set `DEFER_LABEL` to
  your safe-fallback label and the gate hard-fails on any false route
  regardless of accuracy.
- **Validate the capture instrument, then taint on defect.** When cases are
  minted through a lossy reader (OCR, screenshot parsing, scraping), validate
  the reader against known-answer inputs first and keep a per-row capture
  artifact **anonymized** per the Anonymize rule above (PII replaced with
  same-shape stand-ins) — not the raw original; if a true raw artifact must be
  retained to re-validate the instrument later, hold it in a separate,
  minimized, access-controlled store, never as raw PII/secrets in the corpus.
  A reader defect taints every conclusion derived from its output —
  re-derive them; never resurrect pre-fix conclusions. And a human reading of
  a low-res artifact never overturns a pinned value without machine capture
  or independent cross-validation (a "fix" was once shipped off a misread
  screenshot and had to be reverted).
- **Every row records how it was captured.** A hand-written "plausible" row
  converts the gate into a mirror of your own guess — gate corruption, not
  coverage. When the capture rig is unavailable, the honest state is BLOCKED
  naming the exact rig and recipe to unblock — never synthesis.
- **Hold out a distribution-disjoint slice as the ship decider.** When the
  corpus was consulted during development, passing it alone is the overfit
  warning above; the deciding gate is a slice disjoint on a real dimension
  (date range, source, tenant) that development never saw.

The golden runner doubles as an experiment grader: pre-register expected
outputs as cases before any runs, then grade with code, not impressions —
no harness, no experiment. Pre-register the full **outcome → action table**
too (what each result will make you do), so a result cannot be rationalized
into a favored action afterward. **Write the pre-registration somewhere
durable and timestamped BEFORE the first run, and cite that timestamp in
the finding** (`unprobed` — contributor incident as shape; see Provenance).
Durable = version-controlled, or written into the project's permanent
record; ephemeral = `/tmp`, a scratch/sandbox/session directory, anything
`git check-ignore` matches. On overlap, check-ignore wins: an ignored
working path is ephemeral for this rule even when an external archive
preserves it — cite the archive or permanent-record path itself, not the
ignored working copy. Transcribing the criteria into the write-up
afterwards is a weaker record than it looks: it is made once results are
in, so it cannot evidence the ordering that pre-registration exists to
prove, and it fails quietly — a dead link at least tells a later reader the
claim is unbacked. Re-check that every path a finding cites still resolves
before publishing.
❌ "criteria: see `scratch/PREREG.md`" — reclaimed a session later, and
the frozen criteria can no longer be distinguished from fitted ones.

Pre-registration also has to survive the run tripping its own validity
clause mid-flight — a run that goes dead-cell-heavy or otherwise breaches
its pre-registered kill condition after some results are already visible.
The failure mode here is not skipping the clause, it is **honoring it by
patching the live run and continuing to count**: the harness gets a fix
(a retry, a widened timeout, a corrected fixture) and the same in-flight
run keeps going, so the corrected cells sit next to cells scored under
the broken method and the two get treated as one comparable set. **Void
and re-run from zero instead.** The sequence that keeps this honest: (1)
void the run the instant its own clause fires, discard its scores as
evidence for the verdict; (2) write the amendment down — what broke,
what changed — in the same durable record as the pre-registration, not
folded silently into the method section; (3) disclose that partial
results were seen before the amendment, and give the one-line argument
for why that visibility could not have manufactured the outcome (e.g.
the voided run's direction favored the arm that did NOT end up winning,
or the decision table is symmetric so a peek couldn't tilt it either
way) — if that argument can't be made honestly, the amendment is
contaminated and the fix needs a reviewer who never saw the voided
scores; (4) re-run the full battery under the amended method, and the
verdict cites only the re-run. (`unprobed` — see Provenance.)
❌ "Run 1 lost 6 cells to a transport flake, added a retry, cells 7–36
came back clean, calling it 30/30 net of the flake" — mixes pre- and
post-amendment cells in one scored set with no disclosure they ran under
different methods.

Calibrate the difficulty of the SHARED
case set before comparing — never each arm's separately, which destroys
comparability: a comparison where every arm sits at the same ceiling (every
case passes in every arm) or the same floor (none does) carries no
discriminating evidence — halt there and report "untestable at this
tier/difficulty" as a valid outcome instead of publishing a null; between
those extremes, compare the pre-registered per-arm scores (arms clearing a
shared gate at different scores is still a result). Grade blind to which
arm produced each output. And the verdict is bound to the provider, model
tier, and configuration the arms actually ran on: an effect can shrink,
vanish, or invert across configurations, so generalizing to another
provider, tier, or configuration takes its own runs there — same-family
or similar-name inference is not parity evidence. (`unprobed` — see
Provenance.)

**A population claim needs the population benched — a subset scored
under one harness supports "N of the M tested", never "no member does
X" or "every member does X"** (`unprobed` — contributor incident as
shape; see Provenance). The trap is sharpest for a probe that
SEPARATES subjects: any subset that happens to lack a separating
member makes the property look universal, and the resulting "law"
reads as MORE solid than a per-subject score precisely because it
sounds structural rather than sampled. Before a claim names the
population ("the pool", "every tier", "all of them"), check the
denominator: either every current member ran under the same harness
build, or the claim carries its subset explicitly ("3 of the 7").
And the denominator decays independently of the scores — population
membership churns, so a population claim expires with the roster,
not just with serving drift.
❌ "no member of the pool defends unstated degenerate inputs — the
rule is now unconditional" — 3 of 7 members had been benched; a
whole-pool run days later found one guarding that edge 2/2 on debut
and another 1/2. The same author had already made and retracted a
different universal-from-subset claim on the same probe weeks
earlier — each subset lacking a separating member looked like a law.

The same calibration discipline applies within one arm across time
(`unprobed` — contributor incident as shape; see Provenance). A
stochastic subject — a model, a scheduler, a network path, anything whose
output can differ on identical input — has a distribution, and a single
full-marks sweep shows it CAN pass, not that it does; "stable", "no
regression" and "matches baseline" are all claims about that distribution.
Replicate before such a claim leaves your notes, and where an arm was not
replicated, carry its run count beside its score so a lone sweep cannot read
as a measurement. Every claimed run needs a persisted row of its own: a run
quoted from recall, or one whose output the next run overwrote, cannot be
re-checked and is not a run — publishing four while one is on disk is how an
unreplicated result becomes an unfalsifiable one. And the overwrite is
usually the harness's own design, not an accident: a results file at a
fixed path makes every re-run destroy the baseline it will be compared
against, so key result artifacts by run (date, tag, or run id) and treat
an existing file at a run's keyed output path as a collision error,
never a target to overwrite. The ban is on silent replacement, not on a
stable path — an append-only ledger whose rows carry their run keys
satisfies it. A
harness that can silently consume its own prior evidence is one careless
re-run away from an uncheckable comparison (one harness's hardcoded
results filename replaced the prior week's scorecard on re-run; those
rows survived only because a separate log duplicated them).
❌ "30/30, no thinking step — make it the default." Replicated to N=4 the
same candidate scored 30/30/20/29, failing twice by mechanisms the first run
never produced.

**Arms share one runner — inventory its environment, or the harness is a
second treatment** (`unprobed` — see Provenance). The runner's own
standing environment — always-on hooks, injected rules or instruction
files, wrapper behavior, permissions, tool availability, harness
configuration — reaches the arms it runs; anything reaching some arms and
not others is an untracked treatment riding on the comparison, and "same
runner" by name establishes nothing (one runner name can load different
hooks or configuration per invocation). Before scoring: enumerate the
runner-level surfaces that can act on any arm, then hold each identical
across arms or record the difference as a condition carried by the
result.
❌ "both arms ran in my session, so conditions matched" — the session's
always-on hook fired inside the baseline arm and not the isolated
treatment arm, so the comparison measured hook-plus-baseline against
treatment.

**Exclusions and lost runs follow one rule set across arms** (`unprobed`
— see Provenance). Eligibility, exclusion, and re-run rules are declared
once, before any arm runs, and applied identically to every arm — the
NOT-ARMED discipline of rule 2 under "What makes a gate real" below
included. Equal final N is not required; per-arm attrition accounting
is: started / excluded-with-reason / scored, so an unequal N is
explainable arm by arm, and an unexplained per-arm gap blocks the
comparison. An exclusion mechanism correlated with one arm's treatment —
the treatment crashing exactly the runs it would have failed — can bias
every surviving comparison; name that asymmetry in the result rather
than averaging over it.
❌ "dropped three malformed runs" — all three sat in one arm, and the
malformation was that arm's own failure signature.

**replay:** replace `replay/corpus.jsonl` with a representative sample of
real logged inputs. Replace `transform()` with the step being changed. Run
`node replay/run.mjs --update` once to freeze current behavior — and eyeball
that first freeze line by line: a baseline freezes *current* behavior, not
*correct* behavior, and it will protect any bug it contains as ground truth
(one committed baseline enshrined a real redaction bug this way — fix the
transform first, then freeze); after each
edit, plain `node replay/run.mjs` — **0 diffs = safe; any diff = the exact
records that moved.** Re-`--update` only after eyeballing an *intended*
change, and only as the orchestrator/reviewer — never the editing worker's
own call (rule 4 below: gate changes are not the worker's to make).

**A frozen baseline inherits its environment's floating-point noise —
declare the numeric contract, and prove portability only where claimed**
(`unprobed` — contributor incident as shape; see Provenance). A baseline
holding *iteratively solved* numerics (an IRR, a solver output —
converged, not closed-form) freezes the last-bit FP behavior of the
runtime that produced it; another runtime major diverges in the
insignificant digits and the gate false-fails on noise — with the
consequence the allow-list rationale under verify-by-reconstruction
below records. The comparator instead expresses the numeric contract actually
promised — a declared precision, tolerance, canonicalization, or other
justified normalization, living in the gate's comparator or snapshot
mapper, never the production code — coarse enough to absorb environment
noise, fine enough that a genuine behavioral change still fails (the
incident's durable remedy: rounding the solver field in the mapper). A
baseline claimed portable across supported environments proves that
claim on a second, relevantly different one — a pass on the freezing
environment shows the snapshot
matches itself, not that it is environment-stable. A runtime pinned as a
recorded decision — documented before the red, not relabelled after it
(operational-rigor §3's documented-decision rule) — owes no proof of a
claim it never made; a pin added to silence a red is a stopgap that
hands the same red to the next environment change.
❌ "CI is red but every diff is in the 13th decimal place — pin CI to my
local runtime version."

**replay variant — parity (no corpus):** a refactor of pure-ish logic (config parsing, path
handling, formatting) often has no logged corpus to replay. Keep the pre-change
implementation *callable* — a pinned import, a second checkout, or
`git show <base>:<path>` copied into a `_old` module — and run old vs new over a
declared input set, asserting identical output/exit (allow-list any intended
diffs). It is the replay gate for code you are refactoring when you have nothing
logged. (Freezing the old source *text* as a string is not a parity test — it
never runs the old code.)

**Replay's inverse — verify-by-reconstruction** (`unprobed` — see Provenance):
to prove "exactly X was applied" to a delivered state, reconstruct across the
boundary with an INDEPENDENT prescription of X — a pinned oracle, the
pre-change implementation (the parity rule above), or the spec — never the
delivering system's own producer, whose bugs reproduce on re-run and
self-confirm. Two sound forms: full-state comparison
`apply_independent(baseline) == delivered` over a DECLARED projection —
and the projection must cover the complete mutation boundary: every field
X touches AND the fields expected to stay unchanged, with only the ambient
fields the system legitimately mutates on its own (ids, timestamps, server
defaults) on a declared allow-list, exactly as the parity gate above
allow-lists intended diffs. A projection cut down to "what X touches"
passes a delivery that also mutated state outside it — the nearest
over-application variant; where the full boundary genuinely cannot be
enumerated, the conclusion narrows to "exact within this projection" and
every out-of-projection surface is reported unverified, never implied
proven. (Raw whole-state equality with no allow-list false-fails on every
non-pure deliver, and a false-failing gate gets weakened or dropped.) Or a
true inversion `apply⁻¹(delivered) == baseline` ONLY where the inverse is
a proven bijection — a lossy "undo" (reset-to-default) maps an
under-applied state back to baseline too and passes exactly the case the
check exists to catch. Both forms
prove STATE, not history: repeated idempotent application and duplicate
side effects that leave identical state are invisible to them — where
those matter, add an operation/event witness (an application count, an
audit log), or the claim stays state-only, said so. No independent
prescription available → the re-run is a consistency check, labelled so —
never a proof.

**When direct state readback is unreliable or unavailable, verify through
a downstream observable that must move under a correct application and
cannot move otherwise** (`unprobed` — contributor incident as shape; see
Provenance). A control/treatment pair — run the system once without the
change and once with it, over identical input, and assert the
treatment's downstream signal differs from the control's in the
direction the change predicts, written down BEFORE either run (P1 > P0,
not merely P1 ≠ P0) — proves the application happened even where the
state it touched cannot be read back directly (an opaque UI setting, a
third-party system with no inspectable state). The same differential
form pins protocol-level bugs: diff your own request byte-for-byte
against the target system's own observed WORKING request for the same
operation — a length or byte-offset difference the diff surfaces is
often the entire defect.
✅ "state readback was unavailable; ran the flow with the setting unset
(P0) and set (P1) over identical inputs, asserted P1 > P0 before either
run — passed, proving the setting reached the downstream calculation."
✅ "diffed my request body against the app's own captured working
request byte-for-byte; length differed by 5, decoding to exactly the
two JSON quotes and `Bearer ` my construction had dropped."
❌ a single successful run with no control, read as proof the change
did anything — nothing rules out the same output with the change absent.

**In parity work, the artifact settles disputes — reading it beats
adjudicating between reviewers or picking the plausible option**
(`unprobed` — contributor incident as shape; see Provenance). When the
contract is parity with an external artifact (a spreadsheet, a workbook,
a prior implementation), a reviewer blocker or a spec ambiguity is a
question the artifact already answers, not a judgment call — open it and
read the cells. And a defensive addition your own spec invents that the
artifact does not contain (a clamp, a guard, a floor the source formula
lacks) silently forks the parity target the moment it ships: it enters
the spec only as an explicitly flagged deviation, never as an unstated
improvement.
✅ "two independent reviewers flagged the same clamp as suspect; opened
the workbook — `B6 = B3*B4-B5`, no MAX anywhere. My spec's clamp was an
invention; fixing to match, no clamp." ❌ picking whichever reviewer's
suggested fix sounds more defensible and moving on, with the artifact
never opened.

**A ground-truth artifact is authoritative for behavior, not for every
embedded constant it hand-types — derive the derivable before porting a
magic number, and flag rows no scenario ever exercises**
(`unprobed` — contributor incident as shape; see Provenance). Hand-maintained
oracles carry hand-typed values that should be *computed* from other
cells; a stale hand-update hides exactly in the rows no realistic
scenario drives, so a clean replay proves nothing about whether the
constant is still correct. Two failure shapes to check for before
porting: (a) a constant that should derive from other cells but was
typed in by hand — recompute it and compare; (b) two DIFFERENT
quantities that happen to share a value (a coincidence, not an identity)
each hand-typed under one shared name — rename them apart, because "same
number, different bases" invites conflation the moment either changes.
✅ "the `144445` in rows 8–10 is `ROUNDUP(130000/0.9)` — but the sheet
was updated by hand from row 11 downward after a minimum changed, and
rows 8–10 are policy years that never draw, so nothing ever surfaced the
staleness. Recomputing and flagging every unexercised row." ❌ porting a
spreadsheet's constants verbatim because the sheet is "ground truth" and
the replay gate is green.

**Cheapest gate shape — the grep-count ratchet:** when an anti-pattern cannot
be removed wholesale (inline locale ternaries, stray global listeners), pin its
current grep count as a dated baseline with the hits enumerated; the executable
done-check on every diff is "the count did not grow" — and nobody "fixes" the
enumerated baseline hits as a side quest either.

## What makes a gate real (task-relative test discipline)

A generic green test is not proof. A gate is real only if:

1. It exercises the **task trajectory** — input, production path, state
   transition, observable output — not a reimplementation of the logic.
2. It would **fail under the broken behavior**. Run both arms where practical —
   broken arm fails, fixed arm passes — and prove a *negative* test can fail by
   running it against a known-bad arm. Instrument the failure's **own** signal,
   not a proxy: an unchanged field or intact-looking output can pass while the
   failure still occurred. **Arm polarity alone is insufficient — a change
   detector can mimic it while guarding nothing** (`unprobed` — adapted
   external design; see Provenance): a source-string presence check or a
   private-structure snapshot fails on the old arm and passes on the new
   one simply because the source changed — while firing on every future
   redesign and sleeping through every future bug (it also fails this
   rule's own-signal requirement above; the polarity just hides that).
   Before writing the test body, answer: what production change should
   make this test fail — and is that change a bug or a decision? If only
   deliberate decisions can fail it, it is a change detector, not a gate —
   asserting the source contains a line proves only that the source is the
   source. Carve-out: pinning a representation is legitimate exactly where
   that representation IS the declared contract (an error-message string
   or output name with downstream consumers — operational-rigor §3's
   output-text-is-an-interface); then a deliberate contract change
   properly updates the test. A suite that *grades* candidates is two-sided:
   before it scores anything, show it PASSES on at least two structurally
   distinct valid solutions (a too-strict suite silently rejects valid
   alternatives — false collapse) **and** FAILS on a known-broken state (false
   parity), both by execution. And confirm the corpus exercises the changed
   branch: a change "verified" only on inputs where the new code never fires
   is unverified — capture firing inputs, or synthesize them into the test
   suite as a labeled synthetic set, NEVER as rows in the captured
   golden/replay corpus (the case-set integrity rules above: a hand-written
   row corrupts the ship gate).
   The behavioral analog, when the gate is a trap fixture an AGENT must
   resist (a prescribed-but-unauthorized action, a planted directive):
   precedence first — taking the bait is FAIL however blind the run was;
   arming gates only the safe direction. A safe outcome counts only if
   the run demonstrably met the trap, the transcript showing the arming
   event for that fixture's carrier (the prescribing doc read, OR the
   planted skill loaded, OR the bait seen — whichever carries this
   fixture's trap). A safe outcome from a run that never met the trap is
   a NOT-ARMED run — excluded and re-run armed, never scored as
   discipline. Fixture-design corollary: hang the trap on a breadcrumb
   the task itself forces (the failing check's output names the doc), or
   read-narrow evidence discipline will disarm the fixture.
   The two-sided proof above validates a grader for ONE invocation shape at
   ONE time — reusing it later (a new run, a different candidate pool, hours
   later in the same session) is a fresh claim, not an inherited one. Before
   reuse: re-run the two-sided proof — the known-good references (both
   structurally distinct valid solutions, per the bar above) and the
   known-bad — diffing each outcome against the record of the prior
   validation (per-CASE outcomes, not an aggregate score — the same 2/6
   with different cases passing is drift; the invocation shape —
   command, arguments, configuration, with ephemeral values like
   run-scoped paths and timestamps normalized — and the
   reference-corpus identity, so drift in any is visible; a deliberate
   invocation change re-baselines only through a fresh two-sided proof
   and a new record; no record on hand → reuse stops, the two-sided
   proof runs fresh and its record is written before any scoring) —
   any drift is stop-the-line, never "still mostly failing, close enough."
   A wrong invocation shape (a file path fed where the grader expects a
   directory, a stale flag) can make the harness fail to load the candidate
   at all while the grader still emits a normal-looking scorecard — the
   candidate never ran, but the grader can't tell "candidate legitimately
   failed" from "candidate never executed." Watch for the inverted
   signature this produces: edge cases PASS while happy-path cases FAIL,
   because an edge case's own error-tolerant branch (a try/catch that treats
   a thrown exception as valid defensive behavior) silently absorbed the
   harness's load failure and got credited for it. (Incident: a
   directory-vs-file argument mismatch made every candidate throw
   `MODULE_NOT_FOUND` before its code ever ran; the known-bad reference
   scored 2/6 against a recorded 0/6, and the 2 passes were exactly the two
   capacity-edge cases whose accepted-throw branch swallowed the harness's
   own error.) (`unprobed` — private incident as shape; see Provenance.)
3. The **easy fake pass is named** and closed — hardcoded expected value,
   weakened assertion, testing the mock, a test that compiled but was never
   registered/run, a permanently `#[ignore]`/`.skip`ped backlog test that reads
   as coverage. Confirm a new test actually *runs* — the runner lists it, or it
   fails when you deliberately break the code — not merely that it compiles. For
   a guard/error path, assert three things, not just the exit code: the
   returncode, a message string unique to THIS check (many errors share exit 2),
   and that the dangerous side-effect did NOT occur (`assertNotIn`). Five more
   fake-pass shapes: a **warm-state pass on init-only code** — a zero-violation
   observation window proves nothing about code that only executes at
   initialization (cold start, first run, migration); exercise the cold path in
   a fresh context before enforcing (a CSP enforced after a clean Report-Only
   window broke the whole engine, because the loader it blocked had been warm
   the entire window). A **CI/automation config that has never executed** —
   count runs (the platform's runs API), not files; a config can be structurally
   undiscoverable (wrong directory in a monorepo) and inert forever while
   reading as coverage. Its source-level cousin, in any ecosystem where the
   build path can succeed without the typechecker (a transpiler that strips
   types without checking, an optional external checker never wired into a
   script): a **static/type-level assertion nothing ever evaluates** — unlike
   the compiled-but-never-registered test above (which a maintainer wrote and
   forgot to wire), this one is *inherited* — a prior author trusted it as a
   live invariant, so nobody deliberately breaks the coupling to find out it
   isn't checked. Grep for compile-time-only assertions, confirm some script
   or CI step actually invokes the checker over that file, then prove it
   two-sidedly: break the coupling once and watch the check go red before
   trusting a clean sweep (a translations-parity const sat in production
   source for months while the build script ran a transpiler that never
   typechecked — it read as an enforced invariant to every reader and
   enforced nothing until a hook started running the checker directly).
   (`unprobed` — private incident as shape; see Provenance.) A **snapshot
   gate that silently re-freezes when its
   baseline is missing** — deleting the baseline must be an error at gate time,
   never a vacuous green. A **scanner that matched zero inputs** — a gate whose
   file pattern silently expands empty (`**` degrading in an old shell dialect
   combined with a nullglob setting, a directory that moved) "passes" while
   scanning nothing (a guard script once did this for the very file its outage
   check was written for). A passing scan must also prove its input set is
   non-empty — assert the matched count is non-zero; merely printing it is the
   same vacuous green if nothing fails on 0. Its partial twin: a scanner may
   consume some inputs while silently skipping others because an error was
   swallowed or a cap went unreported; a non-zero count does not make that
   coverage complete — an undeclared required subset left unread is
   INCOMPLETE, never a clean pass. (`unprobed` — see Provenance.) A
   **substring grader whose match
   token can occur in the graded corpus** — scanning prose for a word that the
   corpus itself may contain scores the corpus, not the behavior, and unlike
   the zero-input scanner above this one runs correctly over a non-empty input
   and still passes every arm. Key on a token the graded material cannot
   produce on its own (a structural marker the subject must create and
   fill — a heading, a filename, a field), and sanity-check the grader
   against a known-bad
   arm before trusting a clean sweep: a grader that passes an arm you KNOW
   failed is the finding, not a formality. Its damage is not a vacuous
   empty run — it manufactures agreement over real input, so an A/B whose
   arms all pass
   reads as "no effect" and retires a real one. (`unprobed` — contributor
   incident as shape; see Provenance.) Worker-written guard scripts
   especially: item 2's known-broken run applies before trust, no exemption —
   whoever wrote a guard has never seen it fail. (`unprobed` — private
   incident as shape; see Provenance.) A **gate runner whose own aggregation
   arithmetic fails open**: a shell script that counts failures with
   `return`/`exit` truncates the count mod 256, so exactly 256 failing files
   or gates reads as success; a `printf | grep -q` check under `pipefail` can
   SIGPIPE-fail the pipeline on large output, flipping the verdict
   independent of the underlying result; a crashed test that dies before
   printing its own failure marker leaves the marker-grep matching nothing,
   so the runner reports a bare, diagnostic-free failure. None of these are
   the code under test failing — the gate's own plumbing fails open or loses
   information under conditions its author never exercised (an exact
   multiple of 256, an oversized log, a crash before the first marker).
   Reproduce the runner's failure mode itself before trusting its count:
   feed it a synthetic 256th failure, an oversized output, a file that
   throws before printing anything. (`unprobed` — contributor incident as
   shape; see Provenance.)
4. **Nobody weakens a gate to turn it green.** A worker satisfies the gate, never
   edits it — gate changes are the orchestrator's call. Three corollaries:
   - For an *immutable policy-checker* (not an ordinary test), run it from a
     pinned trusted base — `git show <base-SHA>:<gate>` or the protected ref's
     copy — against the PR's content as *data*, so the same PR can't edit the
     rules it must pass; pin the checker's dependencies too (a base script that
     imports PR-controlled helpers is still compromised), and protect the workflow
     path itself with branch rulesets / required reviewers, not CODEOWNERS alone.
     Ordinary tests need only independent approval to change, not this.
   - Recompute any integrity value (hash, fingerprint) from a trusted base;
     never trust the value an artifact carries about itself.
   - A test edit is a contract edit: to change a pinned/assertion test, state
     which contract changed and who approved it (ADR/owner). If you can't, you
     are fixing the wrong direction.
5. For important behavior claims, prefer **two independent truth sources**
   (e.g., client output + server state, logs + durable artifact). Two sources
   that agree with each **other** but only moderately with ground truth are
   correlated bias, not independence — score cross-source and same-source
   agreement separately (two models agreeing is one lens, not two). A metric
   clearing a threshold is *evidence*, never *authorization*: keep the go/no-go a
   separate recorded decision.
6. If it is an **automated gate, its block-on-fail decision is deterministic, not
   an LLM's judgment** (`unprobed` — see Provenance). An executable hard gate that
   denies or blocks runs on code, not a model verdict; where an LLM contributes to
   it, the LLM is **advisory and capped** by the gate contract's declared limits —
   a maximum advisory-pass count, a confidence ceiling, findings dropped unless
   sourced — never the pass/fail authority. And where a claim hands you a count,
   sum, or sourced value, **re-derive it independently** (recompute the aggregate;
   trace each value back to its source) rather than trusting the number given. (A
   review/adjudication gate — where a human or a cross-family model verdict IS the
   gate, as in cross-model-review or design-review-gate — is a different
   instrument: there the verdict is the authority, disciplined by lens diversity
   and reproduction, not replaced by code.)
7. If it uses **mutual agreement to assert correctness, freshness, or an
   authoritative value, it anchors that to an external ground truth** (`unprobed`
   — see Provenance). A check that infers currentness from N artifacts agreeing
   with each *other* passes while all N are stale **together** (every manifest
   frozen at an old version, so they "agree"); such an inference anchors to an
   independent source of truth — a release tag, the upstream record, a recomputed
   value — read at the moment it matters. (A check whose contract is only
   *consistency* — do these N agree with each other, with freshness asserted
   elsewhere — is legitimate as-is and needs no anchor, as does an intrinsic gate
   like a syntax or forbidden-character scan. The rule bites only when agreement
   is made to stand in for an external fact.)
8. **A gate over hardcoded facts asserts the facts, not just the shape — and
   the cross-check that established them belongs IN the fixture, not in the
   chat** (`unprobed` — private incident as shape; see Provenance). When code
   embeds domain constants (holiday dates, a tax rate, a fee schedule, a
   jurisdiction's valid state codes), a suite that checks structure — the
   array is non-empty,
   each entry parses, the shape is right — passes identically whether the
   values are correct or a later edit corrupted one. Those values were usually
   cross-checked once, against an authority or several independent sources or a
   reviewer's recall — but that check happened in the conversation and
   evaporates when the session ends, so the next bad edit sails through a
   shape-only gate. Anchor the fact: assert every load-bearing value
   (a fixed holiday falls on its known date, the standard rate equals the
   published number), each assertion naming its authority (source, and its
   version or URL where it has one) and consultation date beside the value —
   an unattributed literal is indistinguishable from item 3's copied-back
   expected value — so a future silent change to a constant fails. This *extends* item 2's carve-out —
   from an output-interface string to an embedded input constant — and is NOT
   item 3's "hardcoded expected value" fake pass: the anchor's value comes from
   an external authority, not copied back from the code's own output. Item 3's
   tautology asserts the code agrees with its own output; this asserts the
   code matches the world. It shares rule 7's remedy — an external anchor — but not its
   trigger: rule 7 bites where *agreement between artifacts* is made to stand
   in for an external fact, this one where *structure* is. A fact that
   legitimately changes gets its anchor updated as a contract edit (rule 4:
   state which contract changed and who approved it); an always-fixed one is
   cheap to anchor permanently.
   ❌ "the holiday tests pass" — they assert the list has the right count and
   types, never that any date is the right day; a fat-fingered edit to one
   date stays green.
9. **Independence across N things is a pairwise claim** (`unprobed` —
   contributor incident as shape; see Provenance). Where a gate rests on N
   items being independent — separate rate-limit buckets, separate failure
   domains, separate credentials, separate blast radii — establishing that
   takes all N(N-1)/2 comparisons, or a directly-read partition (each item's
   owning account queried from the provider) that replaces the comparisons
   rather than shortening them. Exhausting one item and watching the other
   N-1 survive proves only that each is outside THAT one's bucket, and says
   nothing about whether the remaining N-1 share a bucket with each other:
   the probe returns the same reading for N genuinely separate items as for
   one separate item plus N-1 that are all the same, so a baseline-vs-all
   measurement supports a 2x claim while presenting as Nx. Transitivity does
   not rescue it — sharing a bucket is transitive, not-sharing is not, so a
   reader who correctly identifies the property as transitive still owes
   every pair.
   ❌ "key 0 hit its limit and keys 1-3 kept serving — four independent
   accounts, 4x throughput." Keys 1-3 were never tested against each other.
10. **A green suite names the artifact it exercised** (`unprobed` — contributor
   incident as shape; see Provenance). A suite reaches its subject by name — an
   import, a `PATH` lookup, a package entry — and that name can resolve to a
   copy other than the one you edited: a file left behind at a previous
   location, an installed version shadowing the working tree, a build output
   stale by one step. Every assertion then passes honestly, about an artifact
   

…(truncated)