Monte Carlo & Empirical Application (jbes-data-analysis)
When to trigger
- The asymptotic theory exists but the simulation study is thin or one-sided
- The empirical application is a toy illustration rather than a substantive use
- Reviewers will ask "does the method actually work in finite samples / on real data?"
- You need to choose DGPs, baselines, and an application that show the method's value
Why this matters at JBES
JBES is a methods-with-empirics journal: a contribution is incomplete without finite-sample evidence and a substantive empirical application in microeconomics, macroeconomics, business, or finance. The simulation study is how you demonstrate the asymptotics bite at realistic sample sizes; the application is how you demonstrate clear empirical relevance. Both are evaluated by method experts who will reproduce or interrogate them.
Monte Carlo design
- DGPs that span the conditions: include cases that satisfy your assumptions and cases that stress or violate them (dependence, heavy tails, weak identification, high dimension) so the breakdown frontier is visible.
- Sample-size grid: show size/power/coverage/bias/RMSE converging as n grows.
- Honest baselines: compare against the relevant incumbent method(s) under identical DGPs.
- Reported MC uncertainty: give Monte Carlo standard errors of rejection rates; fix and report seeds; document the number of replications and runtime.
The empirical application
- Use a real, recognizable data set (e.g., FRED-MD macro series, CRSP/Compustat finance, IPUMS micro) that the method's novelty genuinely helps.
- Show the new method changes a substantive conclusion or enables an analysis prior methods could not.
- Report inference appropriate to the data (HAC/cluster/dependence-robust); include diagnostics.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. JBES is a business / economic-statistics venue — reviewers weigh estimator validity and simulation evidence, so pair every estimate with its diagnostics and, where relevant, a Monte-Carlo check.
- Many outcomes / specifications:
romano_wolf (step-down FWER, accounts for
cross-test correlation) or benjamini_hochberg — report the adjusted threshold.
- OVB sensitivity:
oster_delta / sensemakr — the confounder strength that would
overturn the headline.
- Inference:
wild_cluster_bootstrap (few clusters), twoway_cluster / conley.
- Re-fit off one handle:
audit_result(result_id) lists the missing checks and the
exact suggest_function for each — no guessing the battery.
- Exhibits:
etable / did_summary_to_latex from the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive (now actually-run) battery in
the appendix. See the executed chain in the JF execution walkthrough.
Checklist
Anti-patterns
- Simulating only DGPs that flatter the method; hiding breakdown
- Reporting rejection rates with no Monte Carlo standard errors
- A toy application with no substantive empirical payoff (off-scope at JBES)
- Omitting the incumbent baseline, so "improvement" is unquantified
- Cherry-picked sample sizes that mask poor small-n behavior
Referee-pushback patterns on the evidence (venue-specific fixes)
| JBES referee objection |
Fix this skill enforces |
| "Simulation DGPs are unrepresentative." |
Calibrate DGPs to the application's moments — persistence, fat tails, cross-sectional dependence — not iid Gaussian |
| "No comparison to standard alternatives." |
Add the incumbent estimator(s) under identical DGPs in the same tables |
| "The application is a toy." |
Use a substantive macro/finance/micro case where the novelty changes a conclusion |
Worked vignette: validating a new long-horizon forecast test
A hypothetical JBES paper proposes a HAC-robust test of equal long-horizon predictability, validated on FRED-MD inflation forecasts (numbers illustrative). The Monte Carlo calibrates the DGP to FRED-MD persistence (AR root near 0.97) and overlapping-horizon dependence, not iid noise; at n=240 the test holds an illustrative size of 5.4% versus nominal 5%, while the Diebold-Mariano benchmark over-rejects at 9.1% under the same DGP. The application then reverses a borderline DM verdict on whether a factor-augmented model beats the random walk at 12 months — a substantive payoff, not a toy. Calibration anchor (hedged): JBES weights careful simulation and a real application roughly equally; a paper strong on only one axis is exposed.
Evidence pass for Journal of Business & Economic Statistics
Run this as a concrete capability pass. First lock the statistical estimand, identification/simulation evidence, empirical illustration, and reproducibility path; then test whether the manuscript addresses econometrics/statistics reviewers who expect methodological credibility plus a business or economic use case.
- Primary move: Audit unit, comparison, uncertainty, missingness, sensitivity, and reproducibility before making any prose or submission recommendation.
- Decision ledger: return
claim / evidence / blocker / next edit rows so the next pass can patch the manuscript directly.
- Sibling comparison: compare against Journal of Econometrics for theory-heavy methods, Econometric Theory for proof-first work, Quantitative Economics for economics-theory methods; if the neighboring outlet has the stronger audience claim, recommend re-routing before polishing.
- Verification floor: before submission-ready advice, re-open
resources/official-source-map.md for volatile rules and name the one unresolved fact that could change the recommendation.
Output format
【DGPs】favorable + stress regimes covered? [Y/N]
【n grid】asymptotics visible as n grows? [Y/N]
【Baselines】incumbent(s) under identical DGPs? [Y/N]
【MC uncertainty】MC SEs + seeds + reps reported? [Y/N]
【Application】substantive, uses the novelty? [Y/N]
【Next step】jbes-tables-figures
1---2name: jbes-data-analysis3description: Use when building the Monte Carlo evidence and the substantive empirical application for a Journal of Business & Economic Statistics (JBES) methods paper. Designs and audits the simulation study and the real-data analysis; it does not derive the asymptotic theory (see jbes-identification-strategy).4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/jbes-data-analysis6---78# Monte Carlo & Empirical Application (jbes-data-analysis)910## When to trigger1112- The asymptotic theory exists but the simulation study is thin or one-sided13- The empirical application is a toy illustration rather than a substantive use14- Reviewers will ask "does the method actually work in finite samples / on real data?"15- You need to choose DGPs, baselines, and an application that show the method's value1617## Why this matters at JBES1819JBES is a methods-with-empirics journal: a contribution is incomplete without **finite-sample evidence** and a **substantive empirical application** in microeconomics, macroeconomics, business, or finance. The simulation study is how you demonstrate the asymptotics bite at realistic sample sizes; the application is how you demonstrate **clear empirical relevance**. Both are evaluated by method experts who will reproduce or interrogate them.2021## Monte Carlo design2223- **DGPs that span the conditions**: include cases that satisfy your assumptions and cases that stress or violate them (dependence, heavy tails, weak identification, high dimension) so the breakdown frontier is visible.24- **Sample-size grid**: show size/power/coverage/bias/RMSE converging as n grows.25- **Honest baselines**: compare against the relevant incumbent method(s) under identical DGPs.26- **Reported MC uncertainty**: give Monte Carlo standard errors of rejection rates; fix and report seeds; document the number of replications and runtime.2728## The empirical application2930- Use a real, recognizable data set (e.g., FRED-MD macro series, CRSP/Compustat finance, IPUMS micro) that the method's novelty genuinely helps.31- Show the new method changes a substantive conclusion or enables an analysis prior methods could not.32- Report inference appropriate to the data (HAC/cluster/dependence-robust); include diagnostics.3334## Execution bridge (StatsPAI / Stata MCP)3536Run the battery, don't just enumerate it. Full map:37[`execution-with-mcp`](../../../shared-resources/empirical-methods/execution-with-mcp.md). JBES is a business / economic-statistics venue — reviewers weigh estimator validity and simulation evidence, so pair every estimate with its diagnostics and, where relevant, a Monte-Carlo check.3839- **Many outcomes / specifications:** `romano_wolf` (step-down FWER, accounts for40 cross-test correlation) or `benjamini_hochberg` — report the adjusted threshold.41- **OVB sensitivity:** `oster_delta` / `sensemakr` — the confounder strength that would42 overturn the headline.43- **Inference:** `wild_cluster_bootstrap` (few clusters), `twoway_cluster` / `conley`.44- **Re-fit off one handle:** `audit_result(result_id)` lists the missing checks and the45 exact `suggest_function` for each — no guessing the battery.46- **Exhibits:** `etable` / `did_summary_to_latex` from the handle — no retyped numbers.4748Keep the decisive checks in the body and the exhaustive (now actually-run) battery in49the appendix. See the executed chain in the [JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).50## Checklist5152- [ ] DGPs include both favorable and assumption-stressing regimes53- [ ] Sample-size grid displays the asymptotics taking hold54- [ ] Incumbent baselines simulated under identical DGPs55- [ ] Monte Carlo standard errors, seeds, and replication counts reported56- [ ] A substantive real-data application, not a toy illustration57- [ ] The application uses the method's novelty and changes/enables a conclusion58- [ ] All numbers regenerate from code (see jbes-replication-and-data-policy)5960## Anti-patterns6162- Simulating only DGPs that flatter the method; hiding breakdown63- Reporting rejection rates with no Monte Carlo standard errors64- A toy application with no substantive empirical payoff (off-scope at JBES)65- Omitting the incumbent baseline, so "improvement" is unquantified66- Cherry-picked sample sizes that mask poor small-n behavior676869## Referee-pushback patterns on the evidence (venue-specific fixes)7071| JBES referee objection | Fix this skill enforces |72|----|----|73| "Simulation DGPs are unrepresentative." | Calibrate DGPs to the application's moments — persistence, fat tails, cross-sectional dependence — not iid Gaussian |74| "No comparison to standard alternatives." | Add the incumbent estimator(s) under identical DGPs in the same tables |75| "The application is a toy." | Use a substantive macro/finance/micro case where the novelty changes a conclusion |7677## Worked vignette: validating a new long-horizon forecast test7879A hypothetical JBES paper proposes a HAC-robust test of equal long-horizon predictability, validated on FRED-MD inflation forecasts (numbers **illustrative**). The Monte Carlo calibrates the DGP to FRED-MD persistence (AR root near 0.97) and overlapping-horizon dependence, not iid noise; at n=240 the test holds an illustrative size of 5.4% versus nominal 5%, while the Diebold-Mariano benchmark over-rejects at 9.1% under the same DGP. The application then reverses a borderline DM verdict on whether a factor-augmented model beats the random walk at 12 months — a substantive payoff, not a toy. Calibration anchor (hedged): JBES weights careful simulation and a real application roughly equally; a paper strong on only one axis is exposed.8081## Evidence pass for Journal of Business & Economic Statistics8283Run this as a concrete capability pass. First lock the statistical estimand, identification/simulation evidence, empirical illustration, and reproducibility path; then test whether the manuscript addresses econometrics/statistics reviewers who expect methodological credibility plus a business or economic use case.8485- **Primary move:** Audit unit, comparison, uncertainty, missingness, sensitivity, and reproducibility before making any prose or submission recommendation.86- **Decision ledger:** return `claim / evidence / blocker / next edit` rows so the next pass can patch the manuscript directly.87- **Sibling comparison:** compare against Journal of Econometrics for theory-heavy methods, Econometric Theory for proof-first work, Quantitative Economics for economics-theory methods; if the neighboring outlet has the stronger audience claim, recommend re-routing before polishing.88- **Verification floor:** before submission-ready advice, re-open `resources/official-source-map.md` for volatile rules and name the one unresolved fact that could change the recommendation.8990## Output format9192```93【DGPs】favorable + stress regimes covered? [Y/N]94【n grid】asymptotics visible as n grows? [Y/N]95【Baselines】incumbent(s) under identical DGPs? [Y/N]96【MC uncertainty】MC SEs + seeds + reps reported? [Y/N]97【Application】substantive, uses the novelty? [Y/N]98【Next step】jbes-tables-figures99```