Verification Discipline

Verification Discipline

When to use this

Any time you are about to say "this is correct / this passed / this is fine." The enemy of verification is not the obvious error, it is the error that looks correct — checking your own work against itself, verifying only what you built and never what you missed, passing one dimension and declaring the whole thing right. This skill is about making "correct" something you proved, not something you asserted.

General rules

Rule 1: Expected values must come from an independent source (never a value computed by the thing under test)

Verification means comparing the artifact's output against an independent correct answer. That answer cannot be produced by the thing being verified — otherwise you are comparing something to itself, it will always match — a perfectly circular proof of absolutely nothing.

• Where correct answers come from: a published authoritative worked example, a hand calculation, a second independent implementation, or an original baseline that existed before the artifact touched it.
• ✗ Compute once with the engine to make a "reference answer," then compute again with the same engine to compare against it → a fake cross-check.
• If the "correct answer" also depends on an untrusted second party, require that the two are genuinely independent (different origins, neither derived from the other).

Rule 2: Never trust a self-reported "I verified it"

"I checked and it's fine" is not evidence; the output is the evidence. Before claiming a pass, actually run the verification command and paste the key result.

• An agent or tool says "real=0 / all green" → do not take it at face value; look at what entitles it to say that (was a real comparison performed, or is it just asserting?).
• Especially for any design where one stage grades itself → distrust by default, and put an external, objective gate behind it.

Rule 2.5: Verify the system's exit artifact, not an intermediate; report "the system is correct," not "the code is correct"

There is exactly one legitimate definition of correct: enter through the real entry point → the whole system runs → you hold the exact artifact the user receives → it shows zero defects against an independent baseline.

• A green pytest run / green unit tests / a green internal module / a green intermediate file (canonical or temporary) are means, and constitute no claim of correctness at all. They prove some layer of the code is not broken, not that what the system produced is right.
• Never substitute "call the function directly and take its output" for the real entry path — what a function produces ≠ what the system produces, and a correct intermediate ≠ a correct exit artifact (the exit is often reshaped by downstream steps; a reverse-engineered delivered file is rewritten by the source-retention step and may differ from the canonical one).
• Reporting language (hard rule): "N tests green" never goes in the conclusion slot; the conclusion slot may only say "system exit: X correct / Y incorrect." Green tests belong in the "means / supporting evidence" slot. Stating "the code is green" as "the system is correct" is a false claim.
• Sister skill ui-station-delivery: whether a station is usable is decided by a real user walking from entry to finish, not by ticking boxes; here the same principle is pushed onto correctness — only the exit artifact's correctness counts.

Rule 3: Reconcile exactly; the numbers must add up

After acceptance, reconcile: how many items were expected, how many were obtained, where the difference is.

• "baseline N + M new = N+M" — it only counts if it adds up; a mismatch means something went unverified or got swallowed.
• ★List one-time costs and steady-state costs separately; neither may mask the other: for any process that includes a first build or a cache seed, performance numbers are reported and baselined separately for cold (including the one-time cost) and warm (steady state). Taking the warm number as the only baseline hides the cold path's real cost (which really does occur when the period rolls over or the source updates); brandishing the cold total misrepresents steady state. Each gets its own assertion.
• The "zero" in "zero differences" must be explainable: was every cell compared and found equal, or did you just "not see any red"? The latter may mean nothing was compared at all.
• ★Always report two numbers for a baseline, never one: a test suite's "passed count" and its "collected count (including skips)" are two different metrics — report only one, and the next reconciliation against the other metric manufactures a fake contradiction (collect 1391 minus passed 1382 = 9 looks like 9 missing tests, but it is collect minus passed). The formula: always report "passed X + skipped Y (collect X+Y)", and reconcile passed against passed, collect against collect — only like-for-like closes. To find out whether tests were deleted, look at git diff for -def test_; do not infer it from a difference of numbers.
• ★Any suite number that lands in a report, a task description, or a handoff must be tagged with "the settled state of which commit": a number not bound to a commit is a source of drift. A mid-flight snapshot taken during a refactor or fix (still red, new tests not all landed, assertions half-updated) must never go into a document — once it does, the next reconciliation treats it as the baseline and manufactures a fake contradiction (for instance "1416 → 1406, minus 10" was really record drift: that 1406 was a snapshot taken mid-refactor while a certain test module was still red and 7 new tests had not all been added — not a settled state; the settled state was +7 → 1423/0-fail). The formula: when you write a number, write " settled state: passed X + skipped Y (collect X+Y) / fail Z"; mid-flight numbers either do not get written, or get explicitly marked "(mid-refactor snapshot, not a settled state)."

Rule 4: Tier your verification; each tier has its own job and never re-verifies another's

Verification is tiered, and each guarantee lives in exactly one tier:

• Fast screen (seconds, on every change): a health check plus a handful of key baseline spot-checks.
• Targeted (verify what you changed): follow the dependency graph and verify only what this change affected.
• Full (slow, exhaustive, only at merge or major gates, schedulable off-peak): re-verify everything. Day-to-day changes run the targeted tier; the full tier runs once, at the gate — but it does run at the gate, and being slow is not an excuse to skip it. (This covers what correctness each tier is responsible for; for why tiering saves time and how the cost is computed, see the sister skill engineering-economy and its economics of test tiering. They are the correctness side and the cost side of the same thing, and are only complete read together.)

Rule 5: The "should be there and isn't" lens — sweep for what is missing, not just what was built

Most verification checks only whether what was built is correct, and misses what should have been built and wasn't. Ask actively:

• Which situations should be covered but have no test? Which fields or dimensions should exist but are empty?
• What was supposed to be handled automatically but is actually not wired up? Treat "nothing happened" as a result that also needs verifying — silence does not mean correct, it may mean nothing ran at all.

Rule 6: Failure modes always lean to the safe side

When verification cannot decide, lean toward safe, not toward convenient:

• Uncertain → treat it as not passing (check once more); never wave it through as "probably fine."
• Extraction or judgment uncertain → hand it back to a human; never guess a value and fill it in.
• The principle: asking a human one extra time (annoying but safe) always beats automatically doing the wrong thing (effortless but dangerous).

Rule 7: Estimates never satisfy a threshold; only measurements do

Threshold and acceptance decisions recognize only measured numbers. Estimates may be reported (they help planning), but they must be explicitly labeled "estimate, not measured" and must never go in the conclusion slot or be used to claim a threshold is met. "Estimated 4.6 minutes (target met)" is not target met — measuring 4.6 minutes is. Once an estimate leaks into a threshold, every downstream decision built on it is built on sand.

Rule 8: An engineering obstacle must never be dressed up as a verification conclusion

When verification cannot complete (OOM, a broken environment, a broken dependency), the conclusion is "the threshold remains unmet + here is one item to fix" — not "we were blocked, so count it as passed," and not "escalate it for a ruling." An obstacle is work, not an acceptance result; restating "I hit a problem" as "the problem blocked verification, so verification is waived" is where a threshold starts silently loosening. Fix the obstacle, measure the real number, and only then does the threshold move.

Rule 9: Before deleting an "unused check," characterize it as a correct emptiness or a coverage hole

When you find a layer of checks idling on some class of artifact (the check runs but no rule ever matches), do not rush to call it pure waste and delete it. Characterize it first:

• Correct emptiness (by design): that class of artifact has a stronger alternative oracle covering the same risk (byte-for-byte identity replacing item-by-item heuristics) → the emptiness is correct and deleting it is a real saving; but you must be able to name that alternative oracle and say what it covers.
• A coverage hole: the layer is empty because no rules were ever written for that class of artifact → "unused" does not mean "unnecessary," it means this was never verified in the first place, and deleting it ships the hole, sealed, into the release.
• The test: can that artifact vary along that dimension? If it can vary and nobody verifies it, it is a hole; if it cannot vary, or a stronger oracle exists, it is a correct emptiness. Until it is characterized, the words "pure waste" do not hold.

Principles for diffing two artifacts item by item (the values matching ≠ everything matching)

When "verification" means diffing your output against a golden baseline cell by cell or item by item, these principles prevent a false green:

Comparison 1: Align on anchors, never on absolute position

Align the two sides on a stable composite key (sheet name + column header + row key, for instance) and never on absolute row numbers or ordinals — insert one row above or reorder something and every absolute position shifts, producing a pile of fake differences or fake matches. Distinguish identical names with the composite key; do not go by name alone.

Comparison 2: Multiple dimensions, so one dimension cannot hand you a false green

"The values match" does not mean "correct." A single cell has several dimensions that all have to match:

• The value (the computed number) ≠ the formula/source (is this a hardcoded dead value or a live formula?) ≠ the format (number format, style, borders).
• Comparing values only → the formula may have been flattened into a dead value and the format may have drifted while the value happens to match → a false green. If you compare, compare every dimension at once.

Comparison 3: Formulas must stay live; do not read them as values and then compare

When diffing artifacts containing formulas, do not compare their evaluated results as content — that treats a live formula and a dead value of the same number as equal, and lets real degradation (a flattened formula) through. Preserve and compare the formula itself (the source expression), not just the number it happened to produce this time.

Comparison 4: Triage the differences — how the "zero" in "zero differences" is actually established

When differences come out, triage them into three classes rather than treating them all as errors or all as noise:

• Real errors: the artifact's logic is wrong → fix the artifact.
• Artifacts of the comparator: the comparator's own problem (misalignment, dimensions not fully covered, an acceptable format difference counted as a difference) → fix the comparator.
• Accepted: known, harmless differences (a flaw in the source itself, faithfully reflected) → mark them accepted, rather than ignoring them silently. Get the comparator to "complete and trustworthy" first, and only then believe the zero it reports — a zero from an incomplete comparator is a fake synthesis, no better than not comparing at all.

Comparison 5: Equivalence is judged at the content layer; a green won through a coincidence window is a false green

• The equivalence gate sits at the content layer: two artifacts are judged equal by comparing content units byte for byte (an archive part by part; a document node by node), not by the outer container. Container-level differences (packaging timestamps, compression parameters) are not equivalence failures — they are a reproducibility question, raised and fixed separately (fixed timestamps = reproducible builds, a bonus). Do not let them pollute the equivalence judgment, and do not pass off "the shells match" as content equivalence.
• The reason it passed must be "the content is equal," never a coincidence: one equivalence test used to go green because two runs happened to land in the same 2-second timestamp bucket — run it on another day and it goes red. If any comparison has a coincidence window where unequal inputs can also pass, first make the output deterministic (pin every non-content input) and then compare; the test must carry an argument for why the coincidence is impossible, or that green does not count.

Case files from this project (supporting evidence, not required for the general rules)

• Independent expected values (Rule 1): one financial-reporting automation project's iron rule was "a building block's computed output for the example input vs. the example's published answer, real=0; the expected value comes from a real source, is not AI-generated, and is not computed by the block itself"; its evaluator "never compares the engine against the engine's own result," and the genuinely independent second party was the original Excel cache from before injection (computed by Excel, not by this engine).
• Never trust a self-report (Rule 2): the three objective lines of defense do not rely on "the maintainer reads the code" (unreadable = ceremonial), but on the objective hard evidence of authority real=0; the expected value for a web-verified check must come from a worked example on the page, not from recomputation.
• Exact reconciliation (Rule 3): every merge reported "baseline 1341 + 31 new tests = 1372"; the zero in "0 differences cell by cell" was established by comparing against the finished product cell by cell, not by "not seeing red."
• ★Re-measure the reconciliation baseline on the spot; do not trust the last batch's record (a meta lesson): a reconciliation once used the old baseline number recorded in a document and came out 2 short — not contamination, but record drift (the old number was 2 below the truth). The lesson: re-measure the baseline on the spot before starting a batch reconciliation (count the current tests with pytest --collect-only, for instance); never trust "what we wrote down last time." Records go stale; only a fresh measurement is accurate.
• Tiered regression (Rule 4): three tiers (smoke / targeted / full-recipe checkup), with the full suite run once, at the merge gate or the checkup.
• Should be there and isn't (Rule 5): a periodic full checkup exists to sweep for "parts every recipe should have but is missing"; it once surfaced a utility function that had never been registered — reported, not silently patched in, and confirmed by a human.
• Fix the comparator before believing it (Comparison 4): an incomplete comparator (comparing only presence/absence, missing font_color and the column anchors) once produced a fake synthesis; the lesson is to bring the comparator to genuinely complete first (compare borders as style strings, add color, composite-key column anchors), then fix everything in one pass, and only then believe a zero.
• Keep formulas live (Comparison 3): during the batch functionalization, formulas were kept un-flattened (no data_only), and a reverse-engineered file only counted once compared cell by cell against the finished product — guarding against "the value matches but the formula degraded into a dead value."
• An estimate held outside the threshold (Rule 7): during one performance overhaul, the implementer submitted "warm batch ~4.6 minutes (estimated)" hoping to close it out; the reviewer blocked it: estimates do not satisfy thresholds — clear the obstacle, re-measure a real batch, and judge on the real number. In the same round the implementer honestly labeled it "an estimate, not a measurement, not submitted as an achievement" — the correct posture.
• "The one true blocker" put back in its place (Rule 8): a memory peak on a large file got the batch measurement OOM-killed twice, and the report was briefly written as "the one true blocker = the memory peak (it blocks measurement and baselining)" — the reviewer stopped it: an obstacle is an item to fix and the threshold stays red. After the real fix (streaming + on-disk point lookups) the same sheet completed with zero OOM, and only then did the real number land.
• Characterizing the empty check (Rule 9): for one class of byte-for-byte copied output, a heuristic data-check layer was completely empty and was briefly characterized as "pure waste, delete it"; on further questioning it was characterized as a correct emptiness — that class has a stronger oracle (part-by-part byte identity against the baseline) covering the same risk, and it could be named. But the same questioning also dug out a real hole (nobody verified the source's period), and the two were handled separately rather than being swept away together under the word "waste."
• The 2-second bucket false green (Comparison 5): a whole-file hash equivalence test used to go green because two writes happened to land in the same timestamp bucket; during a real batch re-measurement the whole-file hash did not match, yet a part-by-part comparison proved all 25 parts byte-for-byte identical — the difference was purely the packaging timestamp. Resolution: the equivalence gate was renamed to part level (already achieved at that point), reproducible builds via fixed timestamps were raised separately as a bonus, and the implementer disclosed the cause of the false green unprompted.