falsify: try to break the claim before a reviewer does
Principle
The job is to refute the claim, not to confirm it. An agent handed a framework
tends to collect supporting evidence and amplify a wrong premise. In
practice, most refuted claims in multi-agent rounds turn out to be the
orchestrator's own hypotheses. So:
- The proposer first writes the claim as a falsifiable statement. It names exact
quantifiers and parameters, gives a kill switch (the observation that would refute
it), and describes the smallest experiment that could refute it.
- The falsifier's prompt says: "Your primary task is to verify this, not to redo it.
If it is wrong, say exactly where; that is worth more than a confirmation."
- Every attempt must actually run something: a computation, a script or a search.
Report real outputs. Never invent them.
- If no counterexample is found, say "could not refute" and list what was tried. Do not
hide behind vague wording.
Input
A row in CLAIMS.md (ID | claim | proposer | status | falsifier evidence | EVIDENCE refs)
with status open, plus the sources it cites (THEORY.md section, EVIDENCE rows, logs,
code).
Attack order (cheap and decisive first)
Work through all six. Stop early only on a clean refutation.
- Definitions. Are all objects well-defined for every parameter allowed? Does a
symbol mean the same thing in the abstract, the theorem and the experiment? Typical
failures: two different definitions of the same quantity (for example, a padded
length and a polynomial degree written with one symbol); an "element of order r"
that is not guaranteed to exist; a cost measure that counts different operations in
the two arms.
- Quantifiers. Is it "for all" or "there exists"? For all parameters, or only for
those satisfying an unstated side condition? Does the proof use the hypothesis in
the direction stated? Is the claimed injectivity or existence only checked on the
examples? Is a necessary condition presented as sufficient? Write the negation and
try to satisfy it.
- Edge parameters. Try the smallest values (p = 2, 3; n = 1; r = 1, 2), boundary
values (p ≡ 3 mod 4 when the claim needs p ≡ 1; B at the admissibility threshold),
degenerate cases (a trivial group, a zero polynomial, an empty set) and the
largest parameter used in the experiments. Do a small exhaustive search wherever
feasible, for example with
skills/sage-check. One finite counterexample settles
the question.
- Numeric recomputation. Recompute every number in the claim from the raw logs.
Do not use the paper's table or a previous summary. Check:
- that the rows of a table come from the same configuration. Mixing the
fastest-time column from one setting with the fewest-operations column from
another is a classic failure;
- that "best vs best" is really best vs best, and the ratio uses the stated protocol;
- arithmetic inside tables (sums, degrees, ratios, rounding:
round vs ceil can
differ by exactly 1);
- that the same number is identical everywhere it appears: abstract, introduction,
evaluation, appendix;
- that failure modes are attributed correctly. An uncaught exception with a core
dump is not a "graceful noise-budget exhaustion" until the error message says so.
- Hidden baseline unfairness. Different parameters (the library's real preset vs
the one printed in the paper), threads, compiler flags, machine, day or load; a
stale build; a weaker baseline than the strongest available; a cited number never
reproduced locally; offline work moved out of the timed region; a
single-configuration win reported as general. Cross-check with
skills/baseline-pin and skills/bench-protocol.
- Literature pre-emption. Search for the core identity or algorithm under other
names. Check: the target paper's own citations; ePrint from the last 24 months;
follow-ups citing the baseline; the textbook treatment (ask
math-librarian).
A reformulation of known work ("X is exactly Y viewed as a module over Z") downgrades
novelty even when everything is correct. Record the ePrint or DOI. Never cite from
memory; if a reference cannot be verified, write "needs verification".
Verdict taxonomy (write into the CLAIMS.md status column)
| verdict |
meaning |
required evidence |
survived |
every attack in the order above was attempted and none succeeded |
list of attempts, with commands and outputs or the searches run |
weakened |
true only under an extra hypothesis, over a smaller parameter range, with a smaller constant, or with less novelty than claimed |
the precise restated claim, plus the counterexample or recomputation that forced it |
refuted |
a counterexample, a recomputation contradicting the number, or prior art containing the result |
a minimal reproducible counterexample (script plus output) or the exact prior-art citation |
A weakened verdict must include the replacement claim text. The proposer either
adopts it or appeals to pi-orchestrator. Keep refuted rows in the table with
their reason, so no one repeats the dead end.
Output format
| C7 | For every prime p ≡ 1 (mod 4) and every B ≤ B_max(p), map φ_B is injective | theorist | weakened | sage-check exhaustive p<200: first failure p=13, B=2 (script: checks/c7.py, output: results/c7.txt). Holds when 2(|A|+1)B < p (sufficient, not necessary). Replacement: "... if 2(|A|+1)B < p" | E14 |
Also append to the shared errata section (STATE.md → "Errata and rolling updates")
when a verdict changes something other agents rely on. State which source now wins.
Veto
falsifier holds a veto. No claim with status open or refuted may appear in
paper/. A weakened claim may appear only in its restated form. writer checks
CLAIMS.md before each section, and submission-rebuttal re-checks before submission.
Review-brief pattern (for parallel falsifier sub-agents)
When fanning out, give each sub-agent:
- one group of claims (arithmetic and units; performance numbers; artifacts and links;
argument completeness; novelty);
- the verdict labels required on the first line (
REFUTED / CONFIRMED / UNVERIFIABLE);
- a "do NOT do" list naming what the other sub-agents cover;
- an instruction to read the other sub-agents' outputs and explicitly adjudicate any
conflicts.
1---2name: falsify3description: Red-team protocol for trying to refute a research claim (theorem, speed-up, security level, novelty) before anyone else does. Use at the P6 falsification gate, before any number or theorem goes into the paper, and whenever a sub-agent reports a surprisingly good result. Produces a verdict of survived, weakened or refuted, with evidence, in CLAIMS.md.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/falsify6---78# falsify: try to break the claim before a reviewer does910## Principle1112The job is to **refute** the claim, not to confirm it. An agent handed a framework13tends to collect supporting evidence and amplify a wrong premise. In14practice, most refuted claims in multi-agent rounds turn out to be the15*orchestrator's own* hypotheses. So:1617- The proposer first writes the claim as a **falsifiable statement**. It names exact18 quantifiers and parameters, gives a *kill switch* (the observation that would refute19 it), and describes the smallest experiment that could refute it.20- The falsifier's prompt says: "Your primary task is to verify this, not to redo it.21 If it is wrong, say exactly where; that is worth more than a confirmation."22- Every attempt must **actually run** something: a computation, a script or a search.23 Report real outputs. Never invent them.24- If no counterexample is found, say "could not refute" and list what was tried. Do not25 hide behind vague wording.2627## Input2829A row in `CLAIMS.md` (`ID | claim | proposer | status | falsifier evidence | EVIDENCE refs`)30with status `open`, plus the sources it cites (THEORY.md section, EVIDENCE rows, logs,31code).3233## Attack order (cheap and decisive first)3435Work through all six. Stop early only on a clean refutation.36371. **Definitions.** Are all objects well-defined for every parameter allowed? Does a38 symbol mean the same thing in the abstract, the theorem and the experiment? Typical39 failures: two different definitions of the same quantity (for example, a padded40 length and a polynomial degree written with one symbol); an "element of order r"41 that is not guaranteed to exist; a cost measure that counts different operations in42 the two arms.432. **Quantifiers.** Is it "for all" or "there exists"? For all parameters, or only for44 those satisfying an unstated side condition? Does the proof use the hypothesis in45 the direction stated? Is the claimed injectivity or existence only checked on the46 examples? Is a necessary condition presented as sufficient? Write the negation and47 try to satisfy it.483. **Edge parameters.** Try the smallest values (p = 2, 3; n = 1; r = 1, 2), boundary49 values (p ≡ 3 mod 4 when the claim needs p ≡ 1; B at the admissibility threshold),50 degenerate cases (a trivial group, a zero polynomial, an empty set) and the51 *largest* parameter used in the experiments. Do a small exhaustive search wherever52 feasible, for example with `skills/sage-check`. One finite counterexample settles53 the question.544. **Numeric recomputation.** Recompute every number in the claim from the raw logs.55 Do not use the paper's table or a previous summary. Check:56 - that the rows of a table come from the *same* configuration. Mixing the57 fastest-time column from one setting with the fewest-operations column from58 another is a classic failure;59 - that "best vs best" is really best vs best, and the ratio uses the stated protocol;60 - arithmetic inside tables (sums, degrees, ratios, rounding: `round` vs `ceil` can61 differ by exactly 1);62 - that the same number is identical everywhere it appears: abstract, introduction,63 evaluation, appendix;64 - that failure modes are attributed correctly. An uncaught exception with a core65 dump is not a "graceful noise-budget exhaustion" until the error message says so.665. **Hidden baseline unfairness.** Different parameters (the library's real preset vs67 the one printed in the paper), threads, compiler flags, machine, day or load; a68 stale build; a weaker baseline than the strongest available; a cited number never69 reproduced locally; offline work moved out of the timed region; a70 single-configuration win reported as general. Cross-check with71 `skills/baseline-pin` and `skills/bench-protocol`.726. **Literature pre-emption.** Search for the core identity or algorithm under other73 names. Check: the target paper's own citations; ePrint from the last 24 months;74 follow-ups citing the baseline; the textbook treatment (ask `math-librarian`).75 A reformulation of known work ("X is exactly Y viewed as a module over Z") downgrades76 novelty even when everything is correct. Record the ePrint or DOI. Never cite from77 memory; if a reference cannot be verified, write "needs verification".7879## Verdict taxonomy (write into the CLAIMS.md status column)8081| verdict | meaning | required evidence |82|---|---|---|83| `survived` | every attack in the order above was attempted and none succeeded | list of attempts, with commands and outputs or the searches run |84| `weakened` | true only under an extra hypothesis, over a smaller parameter range, with a smaller constant, or with less novelty than claimed | the precise restated claim, plus the counterexample or recomputation that forced it |85| `refuted` | a counterexample, a recomputation contradicting the number, or prior art containing the result | a minimal reproducible counterexample (script plus output) or the exact prior-art citation |8687A `weakened` verdict must include the **replacement claim text**. The proposer either88adopts it or appeals to `pi-orchestrator`. Keep `refuted` rows in the table with89their reason, so no one repeats the dead end.9091## Output format9293```markdown94| C7 | For every prime p ≡ 1 (mod 4) and every B ≤ B_max(p), map φ_B is injective | theorist | weakened | sage-check exhaustive p<200: first failure p=13, B=2 (script: checks/c7.py, output: results/c7.txt). Holds when 2(|A|+1)B < p (sufficient, not necessary). Replacement: "... if 2(|A|+1)B < p" | E14 |95```9697Also append to the shared errata section (`STATE.md` → "Errata and rolling updates")98when a verdict changes something other agents rely on. State which source now wins.99100## Veto101102`falsifier` holds a veto. No claim with status `open` or `refuted` may appear in103`paper/`. A `weakened` claim may appear only in its restated form. `writer` checks104CLAIMS.md before each section, and `submission-rebuttal` re-checks before submission.105106## Review-brief pattern (for parallel falsifier sub-agents)107108When fanning out, give each sub-agent:109110- one group of claims (arithmetic and units; performance numbers; artifacts and links;111 argument completeness; novelty);112- the verdict labels required on the first line (`REFUTED` / `CONFIRMED` / `UNVERIFIABLE`);113- a "do NOT do" list naming what the other sub-agents cover;114- an instruction to read the other sub-agents' outputs and **explicitly adjudicate any115 conflicts**.