Falsifying

Falsifying

Every gate in this repo reports success. None of them proves it could have reported failure.

A gate that cannot fail is not a gate — it is a decoration that costs trust twice: once when it passes something broken, and again when people learn to ignore it. This skill assumes the green light is lying and tries to prove it.

Scope is the apparatus: checkers, gates, coverage numbers, CI jobs, test suites, dashboards. For latent defects in the product itself, use bug-hunter. For abuse paths against a running stack over HTTP, use attack-test.

Technique 1 — make it go red

Take the claim the signal makes, construct the smallest input that must contradict it, and run it. If the signal stays green, the signal is broken.

1. State the claim in one sentence. "This gate fails when unit coverage is below 80%." Vague claims cannot be falsified; sharpen until the claim names a threshold, a set, or a rule.
2. Design the counter-case. The cheapest thing that must trip it: a threshold set above the current value, a fixture with the defect deliberately present, an empty input, a duplicate id, a value one past the boundary.
3. Run it and read the exit code, not the console text. A checker that prints FAIL and exits 0 is worse than one that says nothing.
4. Ask what a false green would look like and test that shape specifically. Most broken gates fail on a category of input, not on a value: matched by a bare name where the name is not unique, trusting a subprocess exit code that is always 0, scanning a file range that stops short.

Real findings from this technique:

• a coverage gate reported PASS at any percentage because it read make's exit code, and that target only printed the number — caught by re-running with the threshold set above the measured value
• an AC tripwire matched criterion ids across the whole suite, so a card with no tests at all reported full coverage off another card's AC-001 — caught by running it against a card whose test files did not exist

Technique 2 — diff every source of the same fact

A fact stated in more than one place is a fact that has already drifted somewhere.

1. Enumerate every source. For an endpoint path: the router, the router test, the e2e specs, the API spec YAML, the generated index, the Bruno collection. Six places, one fact.
2. Diff them pairwise and write the table. Do not stop at the first disagreement — measured once: three sources disagreed with each other in three different ways.
3. Decide which one is the truth from evidence, not from hierarchy. Running code with a passing test that exercises it beats a document that describes it.
4. Check the citations resolve. A reference to a path, a card, a decision, or an upstream contract is a claim; open it. A citation pointing at nothing reads as proof and is not.

Artifacts — no artifact means it did not happen

A falsification pass reports one of exactly two things:

• the counter-case, verbatim, with the output showing the signal going red — or staying green, which is the finding; or
• the source table, listing each place the fact appears and what it says.

"I reviewed it and it looks correct" is not a result. If a signal could not be falsified, say what you tried and why it held — that is the useful half of a negative result.

When to run it

• Mandatory: whenever a gate, checker, or verification script is written or changed. Every defect this skill was built from lived in a gate, not in the product.
• Before quoting a number in an MR or a status report.
• When a suite passes on code nobody has run, or a metric improves without work that explains it.
• When a task is handed to a subagent whose success criterion is "make the gate green" — that brief is satisfiable by an empty test, so audit the result rather than the report.

The advisor gate — mandatory, twice

Calling a gate broken is a claim about someone else's work, and "the counter-case stayed green" is one reading of an output you produced yourself. When the advisor tool is present in the session, consult it at two points: once before you commit to the reading — you have the run output and are about to declare the signal broken, or sound — and once before the result leaves your hands, with the write-up already drafted so it survives the round trip. Consult again when a counter-case will not trip either way, or when you drop one technique for the other.

Report what came back alongside the artifact. If advisor is not in the session, substitute nothing — say in the write-up that no advisor was available and the reading had no outside reader.

Hand-off

A confirmed finding stops here. Fixing it is the BUG flow: diagnosing-bugs for the cause, tdd for the failing repro, then the fix. This skill produces the symptom that flow needs; it does not skip ahead to the patch.

Rationalizations