Principle Report The Disqualifier

Report the disqualifier

A report is not finished when its numbers are correct. It is finished when a reader who disagrees with it knows exactly where to look. Every claim has a disqualifier — one observation that would sink it — and a report that omits its own disqualifier is persuasive in proportion to how wrong it can be.

Four disqualifiers, four sections. Use the ones that apply.

1. What would refute this

Any causal claim states, in the same document, the observation that would falsify it — a measurement, not a hedge. "This may not generalise" is not a falsification section; "if the effect survives when X is held constant, the mechanism is not X, and here is that run" is.

Grade every claim in the report on one visible scale — measured / inferred / conjecture — and put the grade next to the claim, not in a preamble. A reader must be able to strip the report to its measured rows and see what is left.

2. Did the path survive

A summary statistic computed over a path can describe a path that ended. A positive year through an account that was liquidated in March; a mean latency across an interval containing an outage; a mean reward over a trajectory that terminated; an eventual success after a retry budget was exhausted.

Whenever the process has an absorbing state — a barrier it cannot return from — the report carries a survived yes/no column and the date of absorption, beside the final value. The expected value across many runs and the outcome of the one run you get are different quantities, and only the second one is yours.

3. Before and after the fix

When a defect is corrected mid-analysis, report the headline metric under both readings in the same message, with the sign. A table that quietly contains both the pre-fix and post-fix cell has published the reversal without saying it, and the number that travels onward will be whichever one someone quoted first.

4. Which unvaried assumption flips the sign

Every report names its frozen constants — the cost, the threshold, the window, the rate — and, for each one that appears in the metric expression, the value at which the conclusion reverses. If no such value exists inside a plausible range, say so. Disclosure is not this: repeating an assumption twelve times is not the same as testing it once.

An axis you swept that changed no output row is also a finding. Report it as inert rather than presenting its cells as distinct results.

Must never

• Ship a gate that checks a section's heading and calls that a content check. grep -qi "falsification" passes a report that says no falsification was attempted. Assert the property, then prove the gate by deleting the section body and watching it fail — [[principle-assert-invariants-not-last-bug]], step 5.
• Present a headline figure whose cost, fill, or capacity basis is absent. Prefix it PROVISIONAL — until that basis and its sensitivity range are in the same message.
• Let a fan-out end without a terminal aggregation step. A finding stated independently in several sibling reports, and read by nobody across them, is the most expensive kind of undiscovered result.

Grounding

• Falsifiability as the demarcation of a real claim → Karl Popper, The Logic of Scientific Discovery (1959; Ger. 1934).
• Committing to the refuting test before seeing the result → Chris Chambers and Loukia Tzavella, "The past, present and future of Registered Reports," Nature Human Behaviour 6, 29-42 (2022) https://doi.org/10.1038/s41562-021-01193-7.
• Grading each claim on one visible evidence scale → Gordon H. Guyatt et al., "GRADE: an emerging consensus on rating quality of evidence and strength of recommendations," BMJ 336, 924-926 (2008) https://doi.org/10.1136/bmj.39489.470347.AD.
• Why an ensemble average does not describe the single path when the path can be absorbed → Ole Peters, "The ergodicity problem in economics," Nature Physics 15, 1216-1221 (2019) https://doi.org/10.1038/s41567-019-0732-0; J. L. Kelly Jr., "A New Interpretation of Information Rate," Bell System Technical Journal 35(4), 917-926 (1956) https://doi.org/10.1002/j.1538-7305.1956.tb03809.x.
• Reporting the whole space of defensible specifications rather than one → Uri Simonsohn, Joseph P. Simmons, Leif D. Nelson, "Specification curve analysis," Nature Human Behaviour 4, 1208-1214 (2020) https://doi.org/10.1038/s41562-020-0912-z; Andrea Saltelli et al., "Why so many published sensitivity analyses are false," Environmental Modelling & Software 114, 29-39 (2019) https://doi.org/10.1016/j.envsoft.2019.01.012.
• Findings that exist only across sibling reports and are never aggregated → Robert Rosenthal, "The file drawer problem and tolerance for null results," Psychological Bulletin 86(3), 638-641 (1979) https://doi.org/10.1037/0033-2909.86.3.638.
• The "no losing day" / too-good smell test lives in [[principle-no-lookahead]], not here — this skill is about what a report must disclose, that one is about what an evaluation may read.