Falsify
Run a falsification phase at two boundaries: BEFORE writing implementation code (Mode 1, the design) and BEFORE merging (Mode 2, the fix). The goal is to find the one live probe or measurement that would invalidate the work — before it costs a PR cycle or a bad deploy.
The workflow and Step 4 outcomes below are Mode 1. Mode 2 has its own outcomes.
When to run
- Any ticket integrating with an external API (Google Docs/Drive/Calendar, Coda, BigQuery, CloudFront, AWS, OAuth flows)
- Any design resting on claimed system behaviour (query-planner pruning, header forwarding, export formats, ordering guarantees)
- Before writing an ADR for such work
Workflow
Step 1: Enumerate assumptions
List every factual assumption the design depends on. Sources of assumptions:
- API response shapes and export formats
- Ordering / concatenation / pagination guarantees
- Auth scopes and token permissions actually granted
- What a code comment or doc claims vs what the current code does
- Query planner / performance behaviour ("partitioning will prune this")
- Header/metadata forwarding through proxies and CDNs
Step 2: Rank by blast radius
For each assumption: how much work is wasted if it is false? Rank descending. Top 3–5 are "load-bearing".
Step 3: Probe each load-bearing assumption
For each, launch parallel subagents (Agent tool) — one per assumption — tasked to PROVE or DISPROVE it with real evidence:
- A live API call with real credentials (smallest possible: 1 doc, 1 row, dry-run)
- An actual byte/latency/bytes-scanned measurement, not an EXPLAIN guess where a real run is cheap
- Reading the current source code, not comments or docs
Documentation is not proof where a live probe is possible. A probe that can't run (no access, cost) → mark assumption UNVERIFIED, do not silently downgrade to "probably fine".
Step 4: Report
Produce a table:
| Assumption |
Verdict |
Evidence |
Blast radius if wrong |
| ... |
CONFIRMED / DISPROVED / UNVERIFIED |
raw probe output ref |
... |
- Any DISPROVED load-bearing assumption → stop; redesign before code.
- Any UNVERIFIED load-bearing assumption → surface to user; do not implement without explicit go-ahead.
- All CONFIRMED → write the ADR, then implement.
Mode 2 — falsify a fix before merge
Everything above probes the design. This probes the implementation of it, one boundary later. Run it when the diff touches IaC/provider config, deployed env vars, rendered layout, or anything whose behaviour only exists in a running system. One probe inline is fine here — the parallel fan-out in Step 3 is for the design case.
This does not replace verification-before-completion; it sits on top of it. That skill certifies local evidence (tests, lint, build). This one certifies that the change works where it will actually run.
Grade every claim. Only the third counts as verified:
- I reasoned this is true.
- A schema, type or doc says this is true.
- I observed this being true in the running system.
| Diff touches |
Probe |
| IaC / provider config |
Run the provider's dry-run, never apply (pulumi preview, terraform plan). Provider constraints are stricter than the cloud API schema your types validate against — a tree that typechecks can still be rejected at deploy |
| A localhost health check |
Resolve the listening PID and confirm it belongs to this session's process tree; an orphaned server from another worktree is a false green |
| Data / query paths |
Query the real DB or API for the records touched, and paste the rows |
| Rendered layout |
Drive the real UI with the chrome tools, screenshot before/after, and assert the specific computed property you changed |
| "These are/aren't duplicates" |
Diff them and report line/byte similarity — never assert equivalence from reading |
Report the Step 4 table (columns only; its outcomes are for designs), plus an explicit list of anything unprovable here and the access it would need. Mode 2 outcomes:
- Any DISPROVED load-bearing assumption → fix the code, re-probe; the
ship verdict is BLOCKED until it is CONFIRMED.
- Any UNVERIFIED load-bearing assumption → surface to the user; no READY TO MERGE without explicit acceptance of the risk.
- All CONFIRMED → fill in the
Live probe: line of the ship / babysit-pr report with the observed evidence.
Rules
- Mode 1: no implementation code until every load-bearing assumption is CONFIRMED or the user explicitly accepts the risk. Mode 2: no READY TO MERGE under the same condition
- Show raw probe output, not paraphrases
- Probes must be minimal and read-only where possible; never probe with destructive writes against shared/prod resources
- Keep probe scripts in the repo (e.g.
scripts/probes/ or ticket branch) so they can be re-run when the API changes
1---2name: falsify3description: This skill should be used when the user says "falsify this", "red-team this design", "falsify the fix", or "check the assumptions", before implementing integration tickets, before writing an ADR for work touching external APIs, or before merging a change to infrastructure, IaC, deployed config or rendered layout. Enumerates every factual assumption the work depends on, ranks by blast radius, and proves or disproves each with live probes.4metadata:5 source: https://skillmd.com/skills/kjgarza/falsify6---78# Falsify910Run a falsification phase at two boundaries: BEFORE writing implementation code (Mode 1, the design) and BEFORE merging (Mode 2, the fix). The goal is to find the one live probe or measurement that would invalidate the work — before it costs a PR cycle or a bad deploy.1112The workflow and Step 4 outcomes below are Mode 1. Mode 2 has its own outcomes.1314## When to run1516- Any ticket integrating with an external API (Google Docs/Drive/Calendar, Coda, BigQuery, CloudFront, AWS, OAuth flows)17- Any design resting on claimed system behaviour (query-planner pruning, header forwarding, export formats, ordering guarantees)18- Before writing an ADR for such work1920## Workflow2122### Step 1: Enumerate assumptions2324List every factual assumption the design depends on. Sources of assumptions:2526- API response shapes and export formats27- Ordering / concatenation / pagination guarantees28- Auth scopes and token permissions actually granted29- What a code comment or doc claims vs what the current code does30- Query planner / performance behaviour ("partitioning will prune this")31- Header/metadata forwarding through proxies and CDNs3233### Step 2: Rank by blast radius3435For each assumption: how much work is wasted if it is false? Rank descending. Top 3–5 are "load-bearing".3637### Step 3: Probe each load-bearing assumption3839For each, launch parallel subagents (Agent tool) — one per assumption — tasked to PROVE or DISPROVE it with real evidence:4041- A live API call with real credentials (smallest possible: 1 doc, 1 row, dry-run)42- An actual byte/latency/bytes-scanned measurement, not an EXPLAIN guess where a real run is cheap43- Reading the current source code, not comments or docs4445**Documentation is not proof where a live probe is possible.** A probe that can't run (no access, cost) → mark assumption UNVERIFIED, do not silently downgrade to "probably fine".4647### Step 4: Report4849Produce a table:5051| Assumption | Verdict | Evidence | Blast radius if wrong |52|---|---|---|---|53| ... | CONFIRMED / DISPROVED / UNVERIFIED | raw probe output ref | ... |5455- Any DISPROVED load-bearing assumption → stop; redesign before code.56- Any UNVERIFIED load-bearing assumption → surface to user; do not implement without explicit go-ahead.57- All CONFIRMED → write the ADR, then implement.5859## Mode 2 — falsify a fix before merge6061Everything above probes the **design**. This probes the **implementation of it**, one boundary later. Run it when the diff touches IaC/provider config, deployed env vars, rendered layout, or anything whose behaviour only exists in a running system. One probe inline is fine here — the parallel fan-out in Step 3 is for the design case.6263This does not replace `verification-before-completion`; it sits on top of it. That skill certifies local evidence (tests, lint, build). This one certifies that the change works where it will actually run.6465Grade every claim. Only the third counts as verified:66671. *I reasoned this is true.*682. *A schema, type or doc says this is true.*693. **I observed this being true in the running system.**7071| Diff touches | Probe |72|---|---|73| IaC / provider config | Run the provider's dry-run, never apply (`pulumi preview`, `terraform plan`). Provider constraints are stricter than the cloud API schema your types validate against — a tree that typechecks can still be rejected at deploy |74| A localhost health check | Resolve the listening PID and confirm it belongs to this session's process tree; an orphaned server from another worktree is a false green |75| Data / query paths | Query the real DB or API for the records touched, and paste the rows |76| Rendered layout | Drive the real UI with the chrome tools, screenshot before/after, and assert the specific computed property you changed |77| "These are/aren't duplicates" | Diff them and report line/byte similarity — never assert equivalence from reading |7879Report the Step 4 table (columns only; its outcomes are for designs), plus an explicit list of anything unprovable here and the access it would need. Mode 2 outcomes:8081- Any DISPROVED load-bearing assumption → fix the code, re-probe; the `ship` verdict is BLOCKED until it is CONFIRMED.82- Any UNVERIFIED load-bearing assumption → surface to the user; no READY TO MERGE without explicit acceptance of the risk.83- All CONFIRMED → fill in the `Live probe:` line of the `ship` / `babysit-pr` report with the observed evidence.8485## Rules8687- Mode 1: no implementation code until every load-bearing assumption is CONFIRMED or the user explicitly accepts the risk. Mode 2: no READY TO MERGE under the same condition88- Show raw probe output, not paraphrases89- Probes must be minimal and read-only where possible; never probe with destructive writes against shared/prod resources90- Keep probe scripts in the repo (e.g. `scripts/probes/` or ticket branch) so they can be re-run when the API changes