Data Analysis & Robustness (rje-data-analysis)
When to trigger
- Estimates are in hand and you need the robustness suite IO referees demand
- A structural counterfactual needs validation before it goes in the article
- You must report inference correctly for market-level / clustered data
Analysis norms at the IO flagship
RJE referees apply industrial-organization empirical norms. Whether the work is structural or reduced-form, the analysis must show the estimates are credible, well-behaved, and economically sensible, and that counterfactuals are disciplined by the model.
Structural work
- Estimation diagnostics: report objective-function value, convergence, and sensitivity to starting values (non-convex objectives); use multiple starts and report them.
- Economic sanity: elasticities, markups, and marginal costs in plausible ranges; own-price elasticities negative and large enough for positive markups.
- Identification-in-practice: show which moments move which parameters (sensitivity / informativeness of moments).
- Counterfactuals: state maintained assumptions (e.g., fixed product set, conduct unchanged), report them as ranges/bounds, and validate against any out-of-sample episode (a known merger, entry, or price change).
Reduced-form work
- Modern DID where timing is staggered (Callaway–Sant'Anna / Sun–Abraham), with event-study leads and a Goodman-Bacon decomposition.
- Weak-IV-robust inference where instruments are weak; report the first-stage F.
- Placebo / falsification on markets or periods that should not respond.
Inference (both)
- Cluster at the level of treatment/market variation; with few clusters use wild-cluster bootstrap.
- Set and report seeds for simulation, bootstrap, and randomization.
Robustness suite to stage
- Alternative demand specification / functional form (structural) or alternative controls and sample (reduced-form)
- Alternative instruments and conduct assumptions
- Subsample and alternative market-definition checks
- Sensitivity of the headline counterfactual / welfare number to key assumptions
Page-cap discipline
RJE's caps are hard (main text <=40 pp, total <=50 pp). Put the core estimates and one or two decisive robustness exhibits in the main text; move the full robustness battery to the appendix (within the <=10-page appendix+references budget), not into discouraged supporting information.
Diagnostic triage table (structural estimates that IO referees flag)
When a structural estimate looks off, diagnose the symptom first.
| Symptom |
Likely cause |
First check to stage |
| Positive own-price elasticity |
Price endogeneity unhandled or weak instruments |
First-stage strength of cost shifters / BLP instruments |
| Implausibly large markups (>60%) |
Conduct misspecified or marginal cost too low |
Re-estimate under alternative conduct; inspect cost FOCs |
| Estimates jump across starting values |
Non-convex GMM objective, flat ridges |
Multi-start grid; report objective at each start |
| Counterfactual price swings wildly |
Extrapolation outside observed variation |
Bound the counterfactual; restrict to in-support changes |
| Substitution ignores obvious rivals |
Too few random coefficients / no micro-moments |
Add micro-moments or a nesting structure |
Worked vignette: validating a merger simulation
Suppose you estimate random-coefficients logit demand for ready-to-eat cereal, recover marginal costs from Bertrand-Nash FOCs, and simulate a two-brand merger (illustratively, median markup 35%, predicted price rise 4.2% for the merging brands).
- Economic sanity: own-price elasticities near -3.5 are plausible for branded cereal; report that 35% markups sit within the literature's range.
- Identification-in-practice: show cost-shifter instruments move the price coefficient and differentiation instruments move the random-coefficient variances.
- Counterfactual discipline: hold product set and conduct fixed, state it, and report the rise as a range (3.1%-5.4%) across specifications.
- Validation: if a comparable merger occurred nearby, check whether the model predicts its realized price path.
A bare "+4.2%" with no band and no validation invites the first referee pushback below.
Referee-pushback patterns and the venue fix
- "Your counterfactual extrapolates outside observed price variation." Fix: restrict the simulated change to the support of observed prices, or report bounds and flag the extrapolation explicitly.
- "Markups are mechanical artifacts of the conduct assumption." Fix: test conduct where the data allow, or show the markup ranking is robust across Bertrand, Cournot, and partial-collusion assumptions.
- "Inference ignores within-market correlation." Fix: cluster at the market level; with few markets, switch to wild-cluster bootstrap and report the seed.
- "Robustness lives only in a footnote." Fix: stage one decisive robustness exhibit in the main text and route the full battery to the appendix.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. RAND is industrial organization — endogeneity of prices/entry and structural demand; the reduced-form chain for causal claims, structural IO outside it.
- Many outcomes / specifications:
romano_wolf (step-down FWER) or benjamini_hochberg.
- OVB sensitivity:
oster_delta / sensemakr.
- Inference:
wild_cluster_bootstrap (few clusters), twoway_cluster / conley.
- Re-fit off one handle:
audit_result(result_id) lists missing checks + the exact
suggest_function for each.
- Exhibits:
etable / did_summary_to_latex from the handle — no retyped numbers.
Decisive checks in the body, exhaustive battery in the appendix.
JF execution walkthrough.
Anti-patterns
- A single structural run with no starting-value or specification sensitivity
- Counterfactuals reported as point estimates with no assumption bounds
- TWFE on staggered policy timing presented as the headline
- Default robust SEs when variation is at the market level
Output format
【Estimator】structural (demand/conduct/entry/auction) / reduced-form
【Economic sanity】elasticities/markups plausible? [Y/N]
【Robustness done】[specifications, instruments, conduct, subsamples]
【Counterfactual】assumptions stated + bounded? [Y/N]
【Inference】clustering / weak-IV / seeds set? [Y/N]
【Page budget】main robustness in appendix? [Y/N]
【Next step】rje-tables-figures
Source: brycewang-stanford/Awesome-Journal-Skills → RAND-Journal-of-Economics-Skills/skills/rje-data-analysis/SKILL.md
1---2name: rje-data-analysis3description: Use when executing and stress-testing the empirical analysis for a RAND Journal of Economics (RJE) industrial-organization manuscript — estimating structural demand/supply, entry, auction, or reduced-form models, then running the robustness, counterfactual, and inference checks IO referees expect. Analysis discipline, not study design.4---5
6
7# Data Analysis & Robustness (rje-data-analysis)
8
9## When to trigger
10
11- Estimates are in hand and you need the robustness suite IO referees demand
12- A structural counterfactual needs validation before it goes in the article
13- You must report inference correctly for market-level / clustered data
14
15## Analysis norms at the IO flagship
16
17RJE referees apply **industrial-organization empirical norms**. Whether the work is structural or reduced-form, the analysis must show the estimates are **credible, well-behaved, and economically sensible**, and that **counterfactuals are disciplined** by the model.
18
19### Structural work
20- **Estimation diagnostics:** report objective-function value, convergence, and sensitivity to **starting values** (non-convex objectives); use multiple starts and report them.
21- **Economic sanity:** elasticities, markups, and marginal costs in plausible ranges; own-price elasticities negative and large enough for positive markups.
22- **Identification-in-practice:** show which moments move which parameters (sensitivity / informativeness of moments).
23- **Counterfactuals:** state maintained assumptions (e.g., fixed product set, conduct unchanged), report them as ranges/bounds, and validate against any out-of-sample episode (a known merger, entry, or price change).
24
25### Reduced-form work
26- **Modern DID** where timing is staggered (Callaway–Sant'Anna / Sun–Abraham), with event-study leads and a Goodman-Bacon decomposition.
27- **Weak-IV-robust inference** where instruments are weak; report the first-stage F.
28- **Placebo / falsification** on markets or periods that should not respond.
29
30### Inference (both)
31- Cluster at the level of treatment/market variation; with few clusters use **wild-cluster bootstrap**.
32- Set and report **seeds** for simulation, bootstrap, and randomization.
33
34## Robustness suite to stage
35
36- Alternative demand specification / functional form (structural) or alternative controls and sample (reduced-form)
37- Alternative instruments and conduct assumptions
38- Subsample and alternative market-definition checks
39- Sensitivity of the headline **counterfactual / welfare** number to key assumptions
40
41## Page-cap discipline
42
43RJE's caps are hard (main text **<=40 pp**, total **<=50 pp**). Put the core estimates and one or two decisive robustness exhibits in the main text; **move the full robustness battery to the appendix** (within the <=10-page appendix+references budget), not into discouraged supporting information.
44
45## Diagnostic triage table (structural estimates that IO referees flag)
46
47When a structural estimate looks off, diagnose the symptom first.
48
49| Symptom | Likely cause | First check to stage |
50|---|---|---|
51| Positive own-price elasticity | Price endogeneity unhandled or weak instruments | First-stage strength of cost shifters / BLP instruments |
52| Implausibly large markups (>60%) | Conduct misspecified or marginal cost too low | Re-estimate under alternative conduct; inspect cost FOCs |
53| Estimates jump across starting values | Non-convex GMM objective, flat ridges | Multi-start grid; report objective at each start |
54| Counterfactual price swings wildly | Extrapolation outside observed variation | Bound the counterfactual; restrict to in-support changes |
55| Substitution ignores obvious rivals | Too few random coefficients / no micro-moments | Add micro-moments or a nesting structure |
56
57## Worked vignette: validating a merger simulation
58
59Suppose you estimate random-coefficients logit demand for ready-to-eat cereal, recover marginal costs from Bertrand-Nash FOCs, and simulate a two-brand merger (illustratively, median markup 35%, predicted price rise 4.2% for the merging brands).
60
61- **Economic sanity**: own-price elasticities near -3.5 are plausible for branded cereal; report that 35% markups sit within the literature's range.
62- **Identification-in-practice**: show cost-shifter instruments move the price coefficient and differentiation instruments move the random-coefficient variances.
63- **Counterfactual discipline**: hold product set and conduct fixed, state it, and report the rise as a range (3.1%-5.4%) across specifications.
64- **Validation**: if a comparable merger occurred nearby, check whether the model predicts its realized price path.
65
66A bare "+4.2%" with no band and no validation invites the first referee pushback below.
67
68## Referee-pushback patterns and the venue fix
69
70- **"Your counterfactual extrapolates outside observed price variation."** Fix: restrict the simulated change to the support of observed prices, or report bounds and flag the extrapolation explicitly.
71- **"Markups are mechanical artifacts of the conduct assumption."** Fix: test conduct where the data allow, or show the markup ranking is robust across Bertrand, Cournot, and partial-collusion assumptions.
72- **"Inference ignores within-market correlation."** Fix: cluster at the market level; with few markets, switch to wild-cluster bootstrap and report the seed.
73- **"Robustness lives only in a footnote."** Fix: stage one decisive robustness exhibit in the main text and route the full battery to the appendix.
74
75## Execution bridge (StatsPAI / Stata MCP)
76
77Run the battery, don't just enumerate it. Full map:
78[`execution-with-mcp`](../../../shared-resources/empirical-methods/execution-with-mcp.md). RAND is industrial organization — endogeneity of prices/entry and structural demand; the reduced-form chain for causal claims, structural IO outside it.
79
80- **Many outcomes / specifications:** `romano_wolf` (step-down FWER) or `benjamini_hochberg`.
81- **OVB sensitivity:** `oster_delta` / `sensemakr`.
82- **Inference:** `wild_cluster_bootstrap` (few clusters), `twoway_cluster` / `conley`.
83- **Re-fit off one handle:** `audit_result(result_id)` lists missing checks + the exact
84 `suggest_function` for each.
85- **Exhibits:** `etable` / `did_summary_to_latex` from the handle — no retyped numbers.
86
87Decisive checks in the body, exhaustive battery in the appendix.
88[JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).
89## Anti-patterns
90
91- A single structural run with no starting-value or specification sensitivity
92- Counterfactuals reported as point estimates with no assumption bounds
93- TWFE on staggered policy timing presented as the headline
94- Default robust SEs when variation is at the market level
95
96## Output format
97
98```
99【Estimator】structural (demand/conduct/entry/auction) / reduced-form
100【Economic sanity】elasticities/markups plausible? [Y/N]
101【Robustness done】[specifications, instruments, conduct, subsamples]
102【Counterfactual】assumptions stated + bounded? [Y/N]
103【Inference】clustering / weak-IV / seeds set? [Y/N]
104【Page budget】main robustness in appendix? [Y/N]
105【Next step】rje-tables-figures
106```
107
108---
109
110**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `RAND-Journal-of-Economics-Skills/skills/rje-data-analysis/SKILL.md`
thedixitjain