Jebo Robustness

Robustness Strategy (jebo-robustness)

When to trigger

• An experiment has several treatment arms / outcomes and you have not corrected for multiplicity
• A referee could attribute the effect to experimenter demand, confusion, or order effects
• An observational result moves with controls, sample windows, or estimator choice
• An agent-based result might be an artifact of grid, seed, or tuning choices
• You have a long, unstructured "robustness" appendix and no map from check to threat

Organize robustness by behavioral threat, not by appendix list

At JEBO the right question is never "did we run enough checks?" but "for each way this could not be the behavioral mechanism, did we show it survives?" Build a threat → check map. The threats differ sharply across the four archetypes.

Experiments (lab/online/field)

Observational behavioral empirics

• Specification curve / multiverse over reasonable controls and windows; show the headline is not a knife-edge.
• Inference robustness: clustering level, wild-cluster bootstrap with few clusters, randomization inference where natural.
• Placebo / falsification: effect absent where the mechanism predicts none; pre-trend tests for DID.
• Sensitivity to unobservables (Oster δ; Rambachan–Roth honest-DID for parallel-trend violations).

Simulation / agent-based

• Parameter sweeps over the behavioral-rule space; report the region where the result holds.
• Seed sensitivity (many runs, report distribution not one path); grid/step-size invariance.
• Sensitivity of emergent regularities to the behavioral rule chosen (e.g., reinforcement vs. EWA learning).

Distinguish "the effect is real" from "the mechanism is the claimed one"

JEBO's distinctive robustness demand is mechanism robustness: even a real, replicable effect can be driven by a different behavioral channel than claimed. Where possible, add a check that separates your mechanism from the leading alternative (a moderation test the rival channel does not predict, a mediation analysis with the caveats stated, or a treatment that shuts the rival channel off).

Execution bridge (StatsPAI / Stata MCP)

Run the battery, don't just enumerate it. Full map: execution-with-mcp. JEBO spans behavioral/experimental and applied micro; randomization inference for experiments, DiD/IV for observational claims.

• Many outcomes / specifications: romano_wolf (step-down FWER) or benjamini_hochberg.
• OVB sensitivity: oster_delta / sensemakr.
• Inference: wild_cluster_bootstrap (few clusters), twoway_cluster / conley.
• Re-fit off one handle: audit_result(result_id) lists missing checks + the exact suggest_function for each.
• Exhibits: etable / did_summary_to_latex from the handle — no retyped numbers.

Decisive checks in the body, exhaustive battery in the appendix. JF execution walkthrough.

Checklist

• Every robustness exhibit is labeled with the specific threat it neutralizes
• Experiments: demand effects bounded; primary outcome pre-registered; MHT correction reported
• Comprehension/order/subject-pool/attention threats addressed for the relevant design
• Observational: spec-curve + inference robustness + placebo/falsification + unobservables sensitivity
• Simulation: parameter sweeps + seed/grid sensitivity reported
• At least one check separates the claimed mechanism from the leading alternative
• The headline magnitude is reported with honest uncertainty across specifications

Anti-patterns

• A robustness appendix that lists 20 regressions without saying what threat each rebuts
• Reporting only the cell that survives MHT, omitting the corrected p-values across all arms
• Treating "the effect replicates" as proof the mechanism is the claimed one
• Online experiments with no attention/bot screening
• An agent-based headline shown for a single seed and a single grid
• Hand-picked control sets that quietly maximize the coefficient

Output format

【Archetype】experiment / observational / simulation
【Threat → check map】
  - <threat 1> → <check>
  - <threat 2> → <check>
【Multiplicity】primary outcome pre-registered? MHT method:
【Mechanism vs. alternative】<test separating claimed channel from rival>
【Headline stability】<range of estimate across specs + uncertainty>
【Next step】jebo-tables-figures

Source: brycewang-stanford/Awesome-Journal-Skills → Journal-of-Economic-Behavior-and-Organization-Skills/skills/jebo-robustness/SKILL.md