brycewang-stanford Robustness Strategy (ecopol-robustness)
When to trigger
- The headline policy magnitude moves under plausible alternative specifications
- A discussant could attribute the result to one sample, period, or country
- Inference rests on few clusters / few treated units (common in policy evaluations)
- The result is statistically significant but the magnitude — the part a policymaker uses — is imprecise
- You are pre-empting the academic discussant's "have you tried…" before the conference
Robustness is discussant-proofing, not a checklist dump
At EP, robustness has a specific purpose: the paper will be debated live by two invited discussants who act as the referees, and the academic one will arrive with a list of fragility tests. So organize robustness by the threat each test neutralizes, not as a mechanical appendix of "alternative specifications." A reader (and the policy discussant) should see, for each test, which way of being wrong it rules out. Because EP papers carry a recommendation, the central object to defend is the magnitude and its confidence interval, not just the sign.
The threat-to-test map (build the robustness section from this)
| Threat a discussant will raise | The test that answers it |
|---|---|
| "It's driven by one period/sample" | leave-one-out by year/region; pre/post-crisis split |
| "Functional form is doing the work" | alternative specifications; semi/non-parametric check |
| "Parallel trends / pre-trends fail" | event-study leads; honest-DID / pre-trends sensitivity bounds |
| "Confounders you didn't control for" | coefficient-stability / Oster δ bounds; placebo on unaffected groups |
| "Inference is too optimistic with few clusters" | wild-cluster bootstrap; randomization inference |
| "Measurement error in the policy variable" | alternative coding; instrument or bound the error |
| "It won't generalize beyond this case" | replicate on a second jurisdiction/episode if data allow |
Craft moves
- Put a robustness summary in the main text, not buried 40 pages deep: a compact figure or sentence showing the headline number is stable across the threats that matter. The policy discussant will not read the appendix.
- Distinguish robustness from heterogeneity. Stable-everywhere = robust; differs-by-group = a finding to report, not a failure to hide.
- Report the range, not just a star. "The effect lies between X and Y across all specifications" is the EP-useful statement.
- Pre-register the discussant's likely objections (use
ecopol-referee-strategy) and answer the top three in the main text proactively — you cannot reply live as easily as in a referee letter, so anticipate. - Honesty about fragility. If one test moves the number, say what it implies for the policy conclusion rather than hiding it — discussants will find it.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. Economic Policy is policy-facing applied economics; foreground a credible design and a policy-relevant magnitude.
- Many outcomes / specifications:
romano_wolf(step-down FWER) orbenjamini_hochberg. - OVB sensitivity:
oster_delta/sensemakr. - Inference:
wild_cluster_bootstrap(few clusters),twoway_cluster/conley. - Re-fit off one handle:
audit_result(result_id)lists missing checks + the exactsuggest_functionfor each. - Exhibits:
etable/did_summary_to_latexfrom the handle — no retyped numbers.
Decisive checks in the body, exhaustive battery in the appendix. JF execution walkthrough.
Checklist
- Robustness section organized by threat, each test labeled with the threat it neutralizes
- A main-text robustness summary (figure or sentence) — not appendix-only
- Headline magnitude + CI shown to be stable (or its instability honestly bounded)
- Few-cluster / few-treated inference addressed (wild bootstrap / randomization inference)
- Pre-trends / parallel-trends sensitivity reported where the design needs it
- Coefficient-stability or placebo evidence against unobserved confounders
- Heterogeneity reported as a finding, not disguised as robustness
- Top three likely discussant objections answered proactively in the main text
Anti-patterns
- A 20-table appendix of "alternative specifications" with no map to the threats they address
- Defending only significance while the policy-relevant magnitude swings across specifications
- Standard clustered SEs with five treated jurisdictions and no few-cluster correction
- Burying every robustness result in the appendix where the policy discussant never sees it
- Quietly dropping a specification that breaks the result instead of reporting and interpreting it
Output format
【Journal】Economic Policy (EP)
【Skill】ecopol-robustness
【Headline magnitude】X (CI: [.,.]) — stable across threats? Y/N
【Threats neutralized】[sample / functional form / pre-trends / confounders / inference / measurement / external]
【Main-text summary】figure/sentence present? Y/N
【Few-cluster inference】method used
【Honest fragility note】what moves the number and the policy implication
【Next skill】ecopol-tables-figures