# Principle Report The Disqualifier > Apply when writing or gating any report that makes a causal claim or summarises a cumulative outcome — a backtest, an experiment write-up, a capacity or cost model, a postmortem, an eval summary, a benchmark table. The report must carry what would change the reader's decision: what observation would refute the claim, whether the path crossed a state it could not return from, what the headline was before and after the fix, and which unvaried assumption flips the sign. - Skill: `edbertchan/principle-report-the-disqualifier` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add edbertchan/principle-report-the-disqualifier` - Raw SKILL.md: https://api.skillmd.com/api/skills/edbertchan/principle-report-the-disqualifier/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: EdbertChan (https://skillmd.com/u/edbertchan) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/edbertchan/principle-report-the-disqualifier --- # Report the disqualifier A report is not finished when its numbers are correct. It is finished when a reader who disagrees with it knows exactly where to look. Every claim has a disqualifier — one observation that would sink it — and a report that omits its own disqualifier is persuasive in proportion to how wrong it can be. Four disqualifiers, four sections. Use the ones that apply. ## 1. What would refute this Any causal claim states, in the same document, the observation that would falsify it — a measurement, not a hedge. "This may not generalise" is not a falsification section; "if the effect survives when X is held constant, the mechanism is not X, and here is that run" is. Grade every claim in the report on one visible scale — **measured / inferred / conjecture** — and put the grade next to the claim, not in a preamble. A reader must be able to strip the report to its measured rows and see what is left. ## 2. Did the path survive A summary statistic computed over a path can describe a path that ended. A positive year through an account that was liquidated in March; a mean latency across an interval containing an outage; a mean reward over a trajectory that terminated; an eventual success after a retry budget was exhausted. Whenever the process has an **absorbing state** — a barrier it cannot return from — the report carries a survived yes/no column and the date of absorption, beside the final value. The expected value across many runs and the outcome of the one run you get are different quantities, and only the second one is yours. ## 3. Before and after the fix When a defect is corrected mid-analysis, report the headline metric under both readings in the same message, with the sign. A table that quietly contains both the pre-fix and post-fix cell has published the reversal without saying it, and the number that travels onward will be whichever one someone quoted first. ## 4. Which unvaried assumption flips the sign Every report names its frozen constants — the cost, the threshold, the window, the rate — and, for each one that appears in the metric expression, the value at which the conclusion reverses. If no such value exists inside a plausible range, say so. Disclosure is not this: repeating an assumption twelve times is not the same as testing it once. An axis you swept that changed no output row is also a finding. Report it as inert rather than presenting its cells as distinct results. ## Must never - Ship a gate that checks a section's *heading* and calls that a content check. `grep -qi "falsification"` passes a report that says no falsification was attempted. Assert the property, then prove the gate by deleting the section body and watching it fail — [[principle-assert-invariants-not-last-bug]], step 5. - Present a headline figure whose cost, fill, or capacity basis is absent. Prefix it `PROVISIONAL —` until that basis and its sensitivity range are in the same message. - Let a fan-out end without a terminal aggregation step. A finding stated independently in several sibling reports, and read by nobody across them, is the most expensive kind of undiscovered result. ## Grounding - Falsifiability as the demarcation of a real claim → Karl Popper, *The Logic of Scientific Discovery* (1959; Ger. 1934). - Committing to the refuting test *before* seeing the result → Chris Chambers and Loukia Tzavella, "The past, present and future of Registered Reports," *Nature Human Behaviour* 6, 29-42 (2022) . - Grading each claim on one visible evidence scale → Gordon H. Guyatt et al., "GRADE: an emerging consensus on rating quality of evidence and strength of recommendations," *BMJ* 336, 924-926 (2008) . - Why an ensemble average does not describe the single path when the path can be absorbed → Ole Peters, "The ergodicity problem in economics," *Nature Physics* 15, 1216-1221 (2019) ; J. L. Kelly Jr., "A New Interpretation of Information Rate," *Bell System Technical Journal* 35(4), 917-926 (1956) . - Reporting the whole space of defensible specifications rather than one → Uri Simonsohn, Joseph P. Simmons, Leif D. Nelson, "Specification curve analysis," *Nature Human Behaviour* 4, 1208-1214 (2020) ; Andrea Saltelli et al., "Why so many published sensitivity analyses are false," *Environmental Modelling & Software* 114, 29-39 (2019) . - Findings that exist only across sibling reports and are never aggregated → Robert Rosenthal, "The file drawer problem and tolerance for null results," *Psychological Bulletin* 86(3), 638-641 (1979) . - The "no losing day" / too-good smell test lives in [[principle-no-lookahead]], not here — this skill is about what a report must disclose, that one is about what an evaluation may read.