brycewang-stanford Robustness Suite (aeja-robustness)
When to trigger
- The main estimate is in hand and you need to show it is not an artifact of one specification
- A referee asks "is this robust to [alternative controls / sample / functional form / inference]?"
- The result depends on a bandwidth, a clustering choice, or a sample-selection rule that could be questioned
- You suspect specification-search concerns and want to pre-empt them
The AEJ: Applied robustness bar
AEJ: Applied referees probe whether the headline number is stable, honestly inferred, and not the product of researcher degrees of freedom. Robustness here is not a wall of regressions — it is a targeted set of checks each tied to a specific threat to the design. Map every plausible objection to the one check that answers it, and report the checks so the reader sees the estimate barely moves.
| Threat to the result | The check that answers it |
|---|---|
| Omitted confounders | Oster δ / coefficient-stability bounds; added controls in steps |
| Specification search | a specification curve / multiverse; pre-registered primary spec |
| Functional form | levels vs logs, alternative outcome definitions, nonparametric version |
| Sample selection | drop influential units, alternative inclusion rules, balanced vs unbalanced panel |
| Inference too narrow | clustered SEs at the right level, wild-cluster bootstrap (few clusters), randomization inference |
| Design-specific fragility | DID: honest-DID bounds; RD: bandwidth/donut; IV: weak-IV-robust set |
| Multiple outcomes/subgroups | Romano–Wolf / List–Shaikh–Wooldridge MHT adjustment |
Robustness craft
- Lock the primary specification first. Everything else is a perturbation around it; do not present five co-equal specs and let the reader guess which is preferred.
- One threat → one check. A robustness table should read as "here is the worry, here is the evidence it is not a problem."
- Show stability, not just significance. The persuasive object is that the point estimate barely moves, not that it stays starred.
- Be honest about where it weakens. A check that shifts the estimate is information; report it and bound the implication rather than hiding it.
- Match inference to the data structure (clustering, spatial dependence, few clusters) — wrong SEs are the most common AEJ: Applied robustness failure.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. AEJ: Applied is applied microeconomics — labor, health, education, and development field settings where a clean research design is the entry ticket.
- Many outcomes / specifications:
romano_wolf(step-down FWER, accounts for cross-test correlation) orbenjamini_hochberg— report the adjusted threshold. - OVB sensitivity:
oster_delta/sensemakr— the confounder strength that would overturn the headline. - Inference:
wild_cluster_bootstrap(few clusters),twoway_cluster/conley. - Re-fit off one handle:
audit_result(result_id)lists the missing checks and the exactsuggest_functionfor each — no guessing the battery. - Exhibits:
etable/did_summary_to_latexfrom the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive (now actually-run) battery in the appendix. See the executed chain in the JF execution walkthrough.
Checklist
- Primary specification declared (ideally pre-registered) before perturbations
- Each robustness check mapped to a specific threat, not added for volume
- Coefficient-stability evidence (Oster δ or stepwise controls) for selection-on-unobservables
- Inference stress-tested: correct clustering level + wild-cluster/randomization inference where relevant
- Design-specific sensitivity included (honest-DID / RD bandwidth / weak-IV set)
- Multiple-hypothesis adjustment if many outcomes/subgroups
- Stability of the point estimate shown, and any check that moves it reported honestly
Anti-patterns
- A 20-column robustness table with no map from check to threat ("kitchen-sink robustness")
- Reporting only that significance survives while the point estimate wanders
- Hiding the specification that breaks the result
- Clustering at the wrong level or ignoring few-cluster bias, then claiming robustness
- Treating "added more controls and it survived" as sufficient for selection on unobservables
- Subgroup p-hacking with no MHT correction
Worked vignette (illustrative)
An IV estimate of the return to a training program is 0.11 (s.e. 0.04). The robustness suite: (i) effective F of 23 rules out weak instruments; (ii) the Anderson–Rubin 95% set is [0.04, 0.19], so inference is not weak-IV-fragile; (iii) Oster δ implies selection on unobservables would need to be 1.8× selection on observables to nullify it; (iv) wild-cluster bootstrap with 14 clusters keeps the CI away from zero; (v) dropping the largest region moves the estimate to 0.10. The point estimate barely moves — the AEJ: Applied target.
Referee pushback mapped to the robustness fix
- "This looks like specification search." → Declare the pre-registered or primary spec; show a specification curve in which the point estimate barely moves.
- "Did you cluster correctly?" → Cluster at the assignment level; with few clusters report a wild-cluster bootstrap or randomization-inference p-value.
- "Could selection on unobservables explain this?" → Report Oster δ; state how strong selection on unobservables would have to be (relative to observables) to nullify the result.
Output format
【Primary spec】declared / pre-registered? [Y/N] — estimate: ___ (s.e. ___)
【Threat → check map】selection: ___ | spec-search: ___ | form: ___ | sample: ___ | inference: ___ | design: ___
【Inference】clustering level: ___; few-cluster/randomization: ___
【Design sensitivity】honest-DID / RD bandwidth / weak-IV set: ___
【Estimate stability】range across checks: [___, ___]; checks that move it: ___
【Next step】aeja-tables-figures
Source: brycewang-stanford/Awesome-Journal-Skills → AEJ-Applied-Economics-Skills/skills/aeja-robustness/SKILL.md