Smr Simulation Studies

SMR Simulation Studies

Use this to build the Monte Carlo that an SMR reviewer will trust. At a methods journal the simulation is not a formality — it is the primary evidence that the analytical properties hold in finite samples and that the method beats real competitors. A weak or self-serving simulation sinks otherwise sound papers.

Design the DGP space deliberately

Reviewers attack the data-generating process first. Specify it as a designed experiment, not a convenient example:

• Factors and levels: sample size (and, for panels/networks, the relevant dimensions), the parameter that controls the difficulty (effect size, dependence, missingness rate, sparsity), and any nuisance complications. State why each level is realistic for sociological data.
• Coverage of the assumption boundary: include cells where your own assumptions fail, so the paper shows the method's limits, not just its triumphs. SMR rewards honesty about breakdown.
• Calibration to the application: at least one DGP should be calibrated to the real dataset in smr-empirical-illustration, so the simulation speaks to a setting readers care about.
• Replications and seeds: enough Monte Carlo replications for stable estimates of the metrics, with seeds fixed and reported for reproducibility.

The competitor set (non-negotiable)

A simulation that compares the new method only to a naive baseline is the classic reject. Include:

• The current default practitioners actually use.
• The strongest existing alternative for the same problem (often from a neighboring discipline — see smr-literature-positioning).
• Where relevant, an oracle / infeasible benchmark to show the gap your method closes.

If your method loses to a competitor in some cell, report it and explain when each method is preferable — conditional recommendations are more credible than universal victory.

Metrics that match the claim

Coverage and size near the nominal level are the metrics SMR reviewers scrutinize most for inference methods — report the actual numbers, not adjectives.

Presenting the study compactly

• Summarize the full grid in a table or a small-multiples figure; do not narrate every cell.
• Lead with the cell that makes the contribution's point (where the incumbent breaks and the method holds), then show the boundary where the method itself degrades.
• Hand the exhibit design to smr-tables-figures so the grid is self-contained and readable in print.

Checklist

• The DGP is specified as a factorial design with realistic levels, each justified.
• Cells where the method's own assumptions fail are included.
• At least one DGP is calibrated to the empirical illustration's data.
• The competitor set includes the current default and the strongest alternative.
• Metrics match each claim (coverage/size for inference, bias+variance for estimation).
• Replication count and seeds are reported.
• Cells where the method loses are reported with a conditional recommendation.

Anti-patterns

• Strawman comparison: only a naive baseline, never the real competitor.
• Sunny-cell selection: showing only regimes that favor the method.
• Adjective metrics: "good size control" with no rejection rates.
• Cherry-picked n: one favorable sample size with no scaling pattern.
• Uncalibrated fantasy DGP: a design unrelated to any sociological data.
• Hidden seeds / replication count: results that cannot be reproduced.

Output format

[Simulation status] convincing / needs repair / not ready
[DGP factors] <factor : levels, with realism note>
[Competitor set] <default + strongest alternative (+ oracle)>
[Metrics] <bias/RMSE/coverage/size/power/runtime as claimed>
[Boundary cell] <where the method degrades and why that is honest>
[Next SMR skill] smr-empirical-illustration

Source: brycewang-stanford/Awesome-Journal-Skills → Sociological-Methods-and-Research-Skills/skills/smr-simulation-studies/SKILL.md