Challenge

Challenge — does the result survive the choices you didn't make?

A single specification is one draw from a distribution you never looked at.

Why this exists, measured rather than asserted. In a controlled study, 150 autonomous agents were given the same data and the same questions. Effect-size interquartile ranges reached ~10.7 %/yr, and the spread concentrated in discrete measure-choice forks — not in estimation noise. Within a measure family, agents agreed to ~0.25 %/yr. Two findings from that study shape this skill:

• AI peer review left the spread essentially unchanged. Review catches errors; it does not reduce analytical-choice variance. A clean referee report is not robustness.
• Exposure to exemplar papers collapsed the spread by 80–99 % — convergence by imitation, not by correctness. Herding is not agreement.

So the spread has to be measured, not reviewed away.

Preconditions

• A working baseline specification that runs and produces the headline estimate.
• The estimate's estimand stated in words — "the ATT for units treated in 2015, over 2013–2019, on the treated population". If you cannot state it, stop: you cannot challenge a claim you have not defined.
• A fork budget (--forks, default 64). Grid size is the product of your choices; it grows faster than intuition.

Step 1 — Enumerate the forks, before running anything

List every point where a competent, honest analyst could have chosen differently. Do this before seeing any alternative result, and write it down — the list is the pre-registration of the challenge.

Some forks change the estimand, not just the estimate — averaging over them is meaningless. references/fork-catalog.md labels every fork; record estimand forks separately and say so in the report.

Ship --dry-run first. Print the grid size and an estimated runtime before executing anything. A 6-fork grid with 3 options each is 729 fits.

Step 2 — Run the grid

One fit per cell, same seed, same data build. Persist every cell — including failures. A specification that does not converge is information about fragility, not a cell to drop.

Record per cell: the fork coordinates, the point estimate, the standard error, N, and the convergence status.

Step 3 — Report the distribution, not the winner

• Specification curve: estimates sorted, with the fork coordinates shown underneath so a reader can see which choices move the result.
• The share of specifications with the same sign, and the share significant at conventional levels. Report both; they answer different questions.
• Which fork drives the spread. This is the payload. "The result is robust except to the choice between dollar and share volume" is a far more useful sentence than a robustness paragraph.
• Your baseline's percentile in its own distribution. If the headline sits at the 97th percentile of specifications you yourself called defensible, say so.

The descriptive curve is not a test — read it as a description of fragility. If you need inference over the whole curve, use specification-curve analysis's joint permutation test (Simonsohn, Simmons & Nelson 2020), which supplies the sharp null the picture alone lacks.

Step 4 — Attack the identifying assumption

The grid varies what you can vary. The identifying assumption is what you cannot test — so bound it instead, with a named, computable statistic. See references/sensitivity-statistics.md.

Label every statistic executable-here or describe-and-cite. Honesty about what your environment can actually run is itself a verification step; a cited-but-unrun statistic is not evidence.

Step 5 — Placebo and falsification

Where a falsification test exists, run it: a negative-control outcome that should show nothing, a negative-control exposure, a timing placebo. A passed placebo is weak positive evidence; a failed placebo is strong negative evidence. Report both with equal prominence.

Step 6 — Write the ledger entry

Append to the specification-search ledger (verification-ladder.md rung 5): the fork list, the grid size, the distribution summary, which forks moved the result, the sensitivity statistics with their values, and every attempt including the failures.

Pre-commit the interpretation. Before running the grid, write down what result would SUPPORT and what would WEAKEN the claim. The ledger is the arbiter. A robustness exercise interpreted after the fact is not a robustness exercise.

Anti-patterns

• Running until something interesting appears. That is the pathology this skill exists to prevent. Fixed fork list, fixed budget, stated stopping rule.
• Reporting only the specifications that agree — a curve showing only supporting cells is a fishing expedition with better graphics.
• Treating a wide curve as failure. Wide is a finding. Publish it and say what drives it.
• Adding forks nobody would defend to pad the denominator and dilute the fragile cells.

Reference files

Cross-references

• verification-ladder.md — rung 4 (analytic verification) and rung 5 (the ledger)
• /simulation-study — when the question is finite-sample performance, not robustness
• /preregister — reserve a holdout before the search, not after