Aeja Robustness

Robustness Suite (aeja-robustness)

When to trigger

• The main estimate is in hand and you need to show it is not an artifact of one specification
• A referee asks "is this robust to [alternative controls / sample / functional form / inference]?"
• The result depends on a bandwidth, a clustering choice, or a sample-selection rule that could be questioned
• You suspect specification-search concerns and want to pre-empt them

The AEJ: Applied robustness bar

AEJ: Applied referees probe whether the headline number is stable, honestly inferred, and not the product of researcher degrees of freedom. Robustness here is not a wall of regressions — it is a targeted set of checks each tied to a specific threat to the design. Map every plausible objection to the one check that answers it, and report the checks so the reader sees the estimate barely moves.

Robustness craft

1. Lock the primary specification first. Everything else is a perturbation around it; do not present five co-equal specs and let the reader guess which is preferred.
2. One threat → one check. A robustness table should read as "here is the worry, here is the evidence it is not a problem."
3. Show stability, not just significance. The persuasive object is that the point estimate barely moves, not that it stays starred.
4. Be honest about where it weakens. A check that shifts the estimate is information; report it and bound the implication rather than hiding it.
5. Match inference to the data structure (clustering, spatial dependence, few clusters) — wrong SEs are the most common AEJ: Applied robustness failure.

Execution bridge (StatsPAI / Stata MCP)

Run the battery, don't just enumerate it. Full map: execution-with-mcp. AEJ: Applied is applied microeconomics — labor, health, education, and development field settings where a clean research design is the entry ticket.

• Many outcomes / specifications: romano_wolf (step-down FWER, accounts for cross-test correlation) or benjamini_hochberg — report the adjusted threshold.
• OVB sensitivity: oster_delta / sensemakr — the confounder strength that would overturn the headline.
• Inference: wild_cluster_bootstrap (few clusters), twoway_cluster / conley.
• Re-fit off one handle: audit_result(result_id) lists the missing checks and the exact suggest_function for each — no guessing the battery.
• Exhibits: etable / did_summary_to_latex from the handle — no retyped numbers.

Keep the decisive checks in the body and the exhaustive (now actually-run) battery in the appendix. See the executed chain in the JF execution walkthrough.

Checklist

• Primary specification declared (ideally pre-registered) before perturbations
• Each robustness check mapped to a specific threat, not added for volume
• Coefficient-stability evidence (Oster δ or stepwise controls) for selection-on-unobservables
• Inference stress-tested: correct clustering level + wild-cluster/randomization inference where relevant
• Design-specific sensitivity included (honest-DID / RD bandwidth / weak-IV set)
• Multiple-hypothesis adjustment if many outcomes/subgroups
• Stability of the point estimate shown, and any check that moves it reported honestly

Anti-patterns

• A 20-column robustness table with no map from check to threat ("kitchen-sink robustness")
• Reporting only that significance survives while the point estimate wanders
• Hiding the specification that breaks the result
• Clustering at the wrong level or ignoring few-cluster bias, then claiming robustness
• Treating "added more controls and it survived" as sufficient for selection on unobservables
• Subgroup p-hacking with no MHT correction

Worked vignette (illustrative)

An IV estimate of the return to a training program is 0.11 (s.e. 0.04). The robustness suite: (i) effective F of 23 rules out weak instruments; (ii) the Anderson–Rubin 95% set is [0.04, 0.19], so inference is not weak-IV-fragile; (iii) Oster δ implies selection on unobservables would need to be 1.8× selection on observables to nullify it; (iv) wild-cluster bootstrap with 14 clusters keeps the CI away from zero; (v) dropping the largest region moves the estimate to 0.10. The point estimate barely moves — the AEJ: Applied target.

Referee pushback mapped to the robustness fix

• "This looks like specification search." → Declare the pre-registered or primary spec; show a specification curve in which the point estimate barely moves.
• "Did you cluster correctly?" → Cluster at the assignment level; with few clusters report a wild-cluster bootstrap or randomization-inference p-value.
• "Could selection on unobservables explain this?" → Report Oster δ; state how strong selection on unobservables would have to be (relative to observables) to nullify the result.

Output format

【Primary spec】declared / pre-registered? [Y/N] — estimate: ___ (s.e. ___)
【Threat → check map】selection: ___ | spec-search: ___ | form: ___ | sample: ___ | inference: ___ | design: ___
【Inference】clustering level: ___; few-cluster/randomization: ___
【Design sensitivity】honest-DID / RD bandwidth / weak-IV set: ___
【Estimate stability】range across checks: [___, ___]; checks that move it: ___
【Next step】aeja-tables-figures