Theory Simulation

Theory-Simulation — Bridge Theory and Monte Carlo for Statistics Papers

🔬 Model Recommendation: Run this skill on Claude Opus for best results. Designing rate-verifying experiments and reconciling theory with empirical results requires deep reasoning. If your session is not on Opus, run /model opus.

Bridges theoretical results and simulation experiments, two-way:

Theory ←————— stress tests, rate slopes ——————→ Simulation
       ←————— sharper bounds, weakened —————→
              assumptions discovered from sim

Built to top statistics journal standards: clearly stated DGPs, multiple sample sizes, Monte Carlo replications, rate verification via log-log slopes, stress tests on every assumption, finite-sample vs asymptotic comparisons, and publication-grade figures conforming to AoS / JASA / Biometrika / JRSS-B style.

Context: $ARGUMENTS

Pipeline Position

/proofcheck → /proof-repair → /theory-sharpen → /theory-simulation → /proof-writer
   Correct?     Fix issues       Strengthen theory      Verify + stress         Write proofs
                                                        (this skill)

This skill can also run standalone if user has theorems and wants Monte Carlo verification without a full pipeline.

Core Philosophy

A theoretical result is taken seriously by reviewers when simulation:

1. Confirms the predicted rate/coverage/bias under stated assumptions
2. Breaks in the predicted way when assumptions are violated
3. Quantifies the finite-sample regime where asymptotics kick in
4. Reveals improvements (sharper rates, weaker assumptions) for theory iteration

A simulation is taken seriously by reviewers when it has:

• Reproducible DGPs with hierarchical RNG streams (not just a single seed)
• Multiple cells along the asymptotic path the theory uses (e.g., s log d / n fixed, NOT just "multiple n and d")
• MCSE-driven replication count for each metric (NOT a fixed B threshold)
• Honest stress tests, including least-favorable DGPs matched to the theorem
• Inference diagnostics beyond rate: size, local power, interval length, EmpSE vs ModSE calibration
• Publication-grade figures with MC uncertainty shown
• Paired-replicate baseline comparison (all methods on the same synthetic data)
• Failure handling: nonconvergence, singular Hessian, optimizer stalls all logged and reported per cell

References for stat-journal simulation standards: Morris, White & Crowther (2019, Stat in Medicine); Koehler, Brown & Haneuse (2009, Am. Stat.); JASA Reproducibility Editorial (2024).

The Claim Evidence Ledger (the spine)

Everything this skill does is one object: a ledger mapping each theoretical claim to the evidence that does or does not support it. AUDIT mode and DESIGN mode are two entry routes into the same ledger — AUDIT populates it from experiments that already exist, DESIGN populates it as PLANNED and then runs them — and Step 5 reads it. There is one vocabulary, used everywhere in this skill.

Each claim carries a priority: PRIMARY (the paper is built on it), SECONDARY (supporting), PERIPHERAL (decorative). Audit severity is a function of priority and state, not of state alone.

Each claim terminates in exactly one state:

Two axes, not one. Coverage asks whether any experiment aims at the claim. Evidentiary strength asks whether that experiment can actually identify the claim at top-journal standards. Coverage without strength is the most common failure, so a claim covered by an inadequate experiment is PARTIAL[...], never YES.

Reason codes for PARTIAL / CONTRADICTED (multiple allowed, e.g. PARTIAL[grid,precision]):

Severity. PRIMARY + (NO or CONTRADICTED[*]) is CRITICAL. SECONDARY + PARTIAL[reporting] is MINOR. Everything else falls between; judge by how much the paper's central story leans on the claim.

Terminal rule. A ledger with any PRIMARY claim left at NO, CONTRADICTED[*], or PLANNED is not a finished simulation study, whichever mode produced it.

Adequacy Dimensions (both modes)

A simulation is adequate when it holds on five dimensions. AUDIT scores existing experiments against them; DESIGN builds experiments to satisfy them. One contract, two questions — this is what stops the skill from auditing other people's simulations more rigorously than it designs its own.

DESIGN: a planned experiment that has not stated its position on all five is not finished being designed. Design against the reason codes, not merely toward "an experiment exists".

AUDIT: a claim cannot reach YES[*] while any applicable dimension fails. The failing dimension supplies the reason code.

Reuse legitimacy applies only when existing runs are being reused; the other four apply in both modes.

Step 0: Ingest Theory + Existing Code

0A: Locate inputs

Parse $ARGUMENTS:

• A .tex file → read paper, extract theoretical statements
• A paper directory → read paper.tex + any existing simulation code
• A directory with theorems/ or claims/ → read those

0B: Extract simulation-relevant items from theory

Build the Theory-to-Simulation Mapping Table:

| Theory ID | Statement | Quantities to verify | Assumptions to stress |
|-----------|-----------|---------------------|----------------------|
| Thm 1 | √n-consistency under A1-A3 | bias, variance, MSE rate | break A1: i.i.d., A3: tail |
| Thm 2 | Asymptotic normality | coverage of 95% CI | sample size threshold |
| Thm 3 | Rate O(n^{-2/(2+d)}) | log-log slope ≈ -2/(2+d) | d growth, smoothness |
| Cor 1 | Uniform over Θ | sup-norm error vs θ | hardest θ in Θ |

0C: Detect framework axes (from theory-sharpen if available)

Use the 3-axis classification (Data / Framework / Regime) to tailor simulation:

• i.i.d. + parametric + classical → standard n→∞ MC with fixed d
• mixing TS + semiparametric → autoregressive DGP, blocked replications
• i.i.d. + nonparametric + classical → curves at multiple n, smoothness parameter sweep
• i.i.d. + parametric + high-d → vary n and d jointly, sparsity regimes
• online/sequential → cumulative regret, anytime guarantees

0D: Mode detection — DESIGN vs AUDIT

The skill operates in two modes depending on what already exists in the paper:

Auto-detection rules:

• Search the paper for sections matching: "Simulation", "Numerical Experiments", "Monte Carlo", "Empirical Studies", "Numerical Studies", "Numerical Illustration"
• Search for figures with captions mentioning "simulation", "Monte Carlo", "replications", "MSE", "coverage"
• Search for tables containing simulation results (typical header keywords: bias, SE, MSE, coverage, B=, n=)
• If any of the above found → likely AUDIT or HYBRID mode

Ask the user to confirm mode before proceeding. Default to AUDIT when simulation content is detected.

AUDIT MODE — Assess existing simulation results

When the paper already has simulation experiments, evaluate them BEFORE designing new ones. This is the standard situation during paper revision or peer review.

Step A0: Parse the existing simulation section

Extract systematically:

Build papers/<paper-name>/simulation_audit/EXISTING_SIMS.md with the parsed inventory.

Step A1: Map existing sims → theoretical claims (TWO-AXIS Coverage Matrix)

Build a matrix on TWO axes (split per Codex review). A claim can be "covered" by an experiment that doesn't actually identify it — coverage ≠ credibility.

Step A1.0: Claim priority ranking (do this before matrix)

Rank every theoretical claim:

• PRIMARY: central result the paper is built on (main rate, asymptotic distribution, identification)
• SECONDARY: supporting result (consistency, conditions, auxiliary lemma)
• PERIPHERAL: decorative corollary not needed for the main story

Audit severity scales with priority: a gap on a PRIMARY claim is critical; the same gap on a PERIPHERAL one is minor.

Step A1.1: Populate the Claim Evidence Ledger

Fill one ledger row per theoretical claim, using the typed states and reason codes defined in the Claim Evidence Ledger section above. Coverage and evidentiary strength are scored on separate axes: an experiment can exist and still fail to identify the claim, which is why PARTIAL[...] carries reason codes rather than a bare verdict.

A filled example ledger and a worked per-experiment audit: ../stat-shared-references/examples/simulation-audit-example.md.

Step A2: Audit each existing experiment against top-journal standards

For each existing experiment, score against the standards from Steps 1-4:

## Audit: Experiment {k} ({section ref}, "{name}")

### What the experiment does
[DGP with parameters; sample sizes; methods; metric; B; what is reported]

### Audit against standards
| Criterion | Status | Issue |
[asymptotic path declared; loss object matches the theorem; >=6 cells along the path;
MCSE reported; paired replicates; failure rates reported; comparator adequate]

### Ledger effect
Claim {id}: {typed state with reason codes}

A filled example: ../stat-shared-references/examples/simulation-audit-example.md.

Step A2.5: CONTRADICTED protocol (REQUIRED when a claim is tagged CONTRADICTED)

When existing simulation conflicts with theoretical prediction, do NOT just flag "investigate". Run this 7-step structured triage in order:

Step 1. Replication check
  - Rerun the exact cell with the saved seed (if available)
  - Verify the reported numbers reproduce
  - If they don't → reporting / archival error; correct and proceed

Step 2. Metric check
  - Verify the plotted/measured quantity matches the theorem's target
  - Theorem on ‖θ̂ − θ*‖? Paper measures MSE? Compute the correct quantity
  - If contradiction disappears → MISTAKEN COMPARISON, not theorem failure

Step 3. DGP check
  - Verify the DGP actually satisfies the theorem's assumptions
  - Verify the asymptotic path was implemented correctly
  - If contradiction disappears under correct DGP → ASSUMPTION VIOLATED IN SIM, not theorem failure

Step 4. Computation check
  - Inspect convergence failures, optimizer tolerances, numerical issues
  - Inspect tuning behavior (oracle vs data-driven)
  - Look for silent failures dropped from aggregation
  - If contradiction disappears under correct computation → IMPLEMENTATION BUG, not theorem failure

Step 5. MC precision check
  - Compute MCSE for the contradicting quantity
  - Is |empirical − theoretical| > 2 × MCSE?
  - If not → APPARENT CONTRADICTION IS WITHIN MC NOISE; not real

Step 6. Localization
  - Does the contradiction occur in ALL cells, or only specific regimes?
  - Pre-asymptotic only (small n)? → finite-sample effect; theory still valid asymptotically
  - All n? → genuine asymptotic contradiction
  - Off-assumption only? → robustness limit, not theorem failure

Step 7. Escalation routing
  - Implementation / reporting issue (Steps 1, 2, 3, 4) → fix and rerun
  - Within MC noise (Step 5) → not a contradiction; update conclusion to "consistent within MC error"
  - Pre-asymptotic only (Step 6) → reframe as finite-sample regime; consult /proof-repair for FS theorem
  - Off-assumption only (Step 6) → reclassify as robustness failure, NOT theorem failure
  - Survives all 6 checks → GENUINE CONTRADICTION:
      → trigger /proofcheck for proof audit
      → trigger /theory-sharpen for theory revisit
      → flag to user as a major issue requiring author response

Record the triage in simulation_audit/CONTRADICTED_<claim_id>.md for each CONTRADICTED claim. Without this discipline, "contradicted" becomes a dramatic label with no rigor behind it.

Step A2.6: Score every experiment against the adequacy dimensions

For each existing experiment, take a position on all five dimensions from the Adequacy Dimensions section above. A dimension failure maps to its reason code and caps the claim at PARTIAL[...]; a claim reaches YES[*] only when every applicable dimension passes.

Three failures get their own flag in the gap analysis because they routinely sink a submission on their own:

• SELECTION_RISK — evidence of selective reporting (omitted cells, methods, regimes, DGPs, or failures; a single illustrative seed).
• TUNING_GAP — a method advantage shown only under oracle tuning, with no data-driven version.
• COMP_GAP — a paper marketing practicality ("fast", "scalable", "works for large $n$") without runtime, memory, failure-rate, and scaling diagnostics.

Signal-by-signal forensic tables (what to look for, and what each finding implies): ../stat-shared-references/simulation-adequacy-audit.md.

Step A3: Gap analysis — what's missing

Compile gaps in three categories:

A3.1: Claims with NO experimental evidence (most serious)

| Claim | Why it matters | Required new experiment |
|-------|---------------|-----------------------|
| Theorem 3 rate n^{-2/(2+d)} | Main rate result | Design rate-verification along (n,d) grid; ≥6 cells |
| Cor 1 uniformity over Θ | Strengthens Thm 2 from pointwise to uniform | Sup-error experiment over a θ-grid |
| Computational scalability claim | Stated in abstract | Runtime/memory vs n |

A3.2: Experiments with adequacy problems (medium serious)

| Existing experiment | Flaw | Fix |
|---------------------|------|-----|
| Exp 1 MSE vs n | 3 cells, no slope CI | Extend to 6-8 n values; report weighted slope + 95% CI |
| Exp 1 figure | No MC uncertainty | Add MCSE error bars |
| Exp 2 coverage | Single n | Extend to 4-6 n values to show convergence to nominal |
| Exp 3 stress test | Only t_3 | Add t_5 (intermediate), AR(1) (dependence), misspec |

A3.3: Reporting / discipline issues (revision quality)

| Issue | Where | Fix |
|-------|-------|-----|
| No B justification | Throughout | State MCSE target and how B was chosen |
| No paired comparison | Tables | Report paired loss differences method-vs-method |
| Captions lack DGP detail | All figures | Rewrite captions: DGP, n, B, theoretical prediction explicit |
| No failure rates | All cells | Add failure-rate column |
| Missing baselines | Exp 1-2 | Add at least one published competitor + oracle |
| Single θ value | Exp 1-3 | Vary θ at least 3 values, or argue why one is representative |

A3.4: Selection-bias risks (from A2.8)

| Signal | Evidence | Required action |
|--------|----------|----------------|
| SELECTION_RISK: omitted cells | Paper says "additional simulations available on request" | Force inclusion of all cells or explicit exclusion rationale |
| SELECTION_RISK: omitted DGPs | Stress menu has 1 entry | Expand stress menu per theorem-matched least-favorable design |
| SELECTION_RISK: omitted failures | No failure-rate column | Reanalyze raw outputs and report failure rates |

A3.5: Tuning / procedure gaps (from A2.9)

| Gap | Method | Required action |
|-----|--------|----------------|
| TUNING_GAP | Proposed method shown only under oracle λ | Add data-driven λ (CV) and report performance gap |
| TUNING_GAP | CV variability not reported | Add variability over CV folds / random init |

A3.6: Computational adequacy gaps (from A2.10)

| Gap | Where claimed | Required action |
|-----|---------------|----------------|
| COMP_GAP: no runtime | Abstract says "scalable" | Add runtime + memory vs n,d |
| COMP_GAP: no failure rate | Method uses iterative optimization | Add convergence statistics |

Step A4: Targeted improvement plan

Based on the gap analysis, produce a minimal targeted plan — only what's needed to close gaps, not a full redesign:

# Targeted Improvement Plan

## Priority 1: Close critical claim-gaps (NEW experiments)
- E_new1: Verify Theorem 3 rate (along asymptotic path s log d / n = 0.5)
- E_new2: Computational scalability (runtime + peak memory vs n)
- E_new3: Uniformity over Θ for Corollary 1

## Priority 2: Extend / strengthen existing experiments
- Extend Exp 1 to n ∈ {50, 100, 200, 500, 1000, 2000, 5000} with MCSE bars
- Extend Exp 2 coverage to multiple n; add Wilson CIs
- Add t_5 and AR(1) to Exp 3 stress tests

## Priority 3: Reporting + discipline fixes (NO new runs needed)
- Recompute slope estimates with weighted regression + CI
- Rewrite all captions to be content-bearing
- Add paired-difference reporting where possible
- Report B-selection rationale (target MCSE per metric)

## What CAN be reused from existing runs
- Exp 1 raw results: extend rather than rerun if seed/code recoverable
- Exp 2 raw results: extend coverage to new n
- Exp 3 raw results: reuse t_3 cell; add t_5 + AR(1) as new cells

## What MUST be rerun
- If existing code is lost / seeds not stored, baseline experiments must be redone
  with STRICT-tier reproducibility before extension

Write to papers/<paper-name>/simulation_audit/IMPROVEMENT_PLAN.md.

Step A5: Codex cross-audit (if Codex MCP available)

After completing the audit, send it to Codex for independent verification, following ../stat-shared-references/codex-protocol.md. Ask specifically whether any claim the audit marked YES[*] is actually only PARTIAL[...], and whether any gap was missed — the failure mode here is an audit that is too generous to the paper it is auditing.

Reconcile findings explicitly (ACCEPT / PUSH BACK / REQUEST CLARIFICATION with reasoning); a Codex objection is not automatically correct. Record the exchange, and never downgrade or upgrade a ledger state silently to match Codex.

Prompt text: ../stat-shared-references/examples/simulation-codex-review-prompts.md.

Step 1: Design the Simulation Plan (ADEMP-style, claim-based)

Write papers/<paper-name>/simulation/SIMULATION_PLAN.md.

This is the DESIGN entry route into the Claim Evidence Ledger. Open a ledger row for every claim before designing anything, each starting at PLANNED, with its priority set. The design target for a row is the evidence that would move it to YES[strong] — not merely to "an experiment exists", which is what produces PARTIAL[...] rows in audits of other people's papers. Design against the reason codes: pick the grid that defeats [grid], the metric that defeats [metric], the replication count that defeats [precision].

Design by CLAIM, not by theorem. A single theorem typically implies multiple empirical claims (rate, limiting distribution, variance consistency, tuning sensitivity, failure behavior). Each gets its own block. Follow the ADEMP framework (Morris, White & Crowther 2019):

• Aims of the experiment (which specific empirical claim is being tested)
• Data-generating mechanism (DGP, with explicit asymptotic path)
• Estimators / methods compared (including baselines)
• Methods of analysis (metrics, MCSE formulas, summary plots)
• Performance measures (with target MCSE precision)

1A: For each CLAIM, design a verification experiment

## Experiment E{k} — Verify {claim} ({theorem ref})

### Theoretical prediction
[the exact quantity the theorem bounds, with its rate and the assumptions in force]

### Asymptotic path
[which quantities move together, e.g. s log d / n held fixed — not merely "n varies"]

### DGP
[generating mechanism with explicit parameters, and why it is the right difficulty]

### Methods
[proposed estimator + the baselines a referee will expect, run on paired replicates]

### Metrics and MCSE
[the measured quantity, matched to what the theorem bounds; MCSE formula per metric;
target precision that makes the claim identifiable]

### Grid and replication count
[>= 6 cells along the path for a rate claim; B chosen from the target MCSE, not a
round number]

### Adequacy dimensions (all four required; a design missing any is unfinished)
- Truth source: [how theta*/f*/the estimand is defined, and why its own error is far
  below the metric's MCSE]
- Selection discipline: [what will be reported — every cell, method, and regime run,
  including the unflattering ones; state up front that nothing is reported selectively]
- Tuning protocol: [how tuning is chosen, and the data-driven version that must
  accompany any oracle-tuned comparison]
- Computational adequacy: [runtime, memory, failure rate, scaling vs baseline — required
  whenever the paper claims practicality]

### Ledger row
Claim {id} — priority {PRIMARY/SECONDARY/PERIPHERAL} — state PLANNED

A filled example: ../stat-shared-references/examples/simulation-design-example.md.

1B: Stress tests — two-layer design (diagnostic + robustness-claim)

A stress test serves one of two distinct purposes:

Layer 1: Diagnostic (one-at-a-time). For each assumption that is theoretically critical, violate ONLY that assumption. This isolates which assumption matters — a referee needs this to attribute failure.

Layer 2: Robustness-claim (factorial / crossed stress). If the paper claims robustness to multiple violations simultaneously (e.g., heavy tails AND weak signal, misspecification AND dependence), one-at-a-time is NOT sufficient. A method can survive each violation alone yet fail when they co-occur. Run a targeted crossed design — usually a 2×2 or 2×3 factorial of the critical-pair violations, not a full factorial.

Diagnostic stress: candidate menu (NOT a prescription).

These are starting candidates. You must select stressors matched to your theorem's specific assumptions and add theorem-specific least-favorable DGPs.

Required: replace each generic stressor with a theorem-matched version. Example for an M-estimation theorem assuming sub-Gaussian X and bounded influence:

## S1 (diagnostic): violate sub-Gaussian → bounded 4th moment
DGP: X_i ~ t_5 (finite variance + 4th moment ✓, sub-G ✗)
Theoretical prediction: theorem still holds with weaker rate

## S2 (diagnostic): violate sub-Gaussian → infinite variance heavy tail
DGP: X_i ~ t_{1.5} (4th moment fails)
Theoretical prediction: theorem fails

## S3 (least-favorable diagnostic): adversarial contamination at influence point
DGP: clean X_i with prob 1-ε, plus mass at the worst-case point
Theoretical prediction: tests the boundary of the influence-function bound

## S4 (robustness claim): heavy-tailed AND weak signal
DGP: X_i ~ t_5 + ‖θ*‖ shrinks at rate n^{-1/4}
Theoretical prediction: paper's "robustness" claim requires this to hold

1C: Rate verification protocol (mathematically precise)

Step 1: Identify the loss object the theorem bounds.

The slope target depends on WHICH quantity the theorem claims to control:

Force the user to declare the loss object before running. A single skill that defaults to "slope = −2a always" is wrong.

Step 2: Declare the asymptotic path.

If the theorem uses any path parameter (e.g., s log d / n → 0, d/n → γ, nh^β → ∞, signal strength r_n), simulate along that path holding the control parameter fixed. Examples:

Theorem 1: ‖θ̂ − θ*‖² = O_P(s log d / n)
Path: keep s log d / n = 0.5 fixed; vary (n,d,s) so the ratio is preserved
Example grid: (n=200, d=50, s=5), (n=400, d=200, s=10), (n=800, d=800, s=20)

This is the only way to make a slope plot interpretable for high-dim / nuisance theory.

Step 3: Run at ≥6 cells along the path.

• Each cell needs B replications determined by MCSE target (Step 1D below).
• Compute empirical loss at each cell with metric-appropriate MCSE.

Step 4: Multiple slope diagnostics (do NOT rely on a single OLS slope).

(a) Weighted regression of log(empirical loss) on log(n) using inverse MCSE² as weights. Report slope estimate + 95% CI from delta method or bootstrap. (b) Local slopes: compute the slope between each adjacent pair of cells; if these vary systematically with n, asymptotics has not kicked in. (c) Normalized loss plot: plot n^a · empirical loss (or n^{2a} · MSE) versus n. If the rate is correct, this should level off; if it drifts, the rate is wrong or finite-sample bias dominates.

The normalized-loss leveling-off plot is more diagnostic than a single slope number.

Step 5: Pass criteria.

• Weighted slope's 95% CI contains the theoretical value
• Local slopes converge to the theoretical value as n grows
• Normalized loss plot levels off (within a band of ±2× MCSE of its plateau)

If any of these fails, do NOT call the rate "confirmed"; investigate.

1D: Inference diagnostics (REQUIRED for any paper with CIs/tests)

Coverage ALONE is not enough for a top-stat-journal inference paper. A referee will demand at minimum:

MCSE for coverage (Wilson / Jeffreys interval — not sd/√B):

• For B replications and observed coverage p̂: MCSE = √(p̂(1−p̂)/B)
• 95% MC interval for coverage: Wilson interval based on (p̂, B)

MCSE for size (same binomial logic):

• For size α (e.g., 0.05) with B reps: MCSE = √(α(1−α)/B)
• B=1000 → MCSE ≈ 0.0069 for size at 0.05 — non-trivial precision needed

B from MCSE target, NOT from a fixed threshold:

• Choose target MCSE per metric (e.g., MCSE ≤ 0.005 for coverage near 0.95 → B ≥ 1900)
• If a cell shows MCSE too high after initial B, extend with more replicates (preserve seed determinism)
• Document each metric's chosen target MCSE in the plan

References: Morris, White & Crowther (2019); Koehler, Brown & Haneuse (2009); Brown, Cai & DasGupta (2001, Statistical Science).

1D′: Metric-specific MCSE formulas (DO NOT use sd/√B for everything)

1E: Method comparison — REQUIRED to be paired across replicates

Always include ≥1 of:

• Competing method from the literature (e.g., MLE vs proposed estimator)
• Oracle (knows nuisance) — should bound your method's loss
• Naive baseline (e.g., empirical mean)

Paired-replicate rule (NOT optional): all methods compared in a cell must evaluate on the SAME replicate datasets (same RNG stream per replicate). Compare via paired loss differences:

for each cell:
  for each rep:
    data = generate(rng[cell, rep], n, d, dgp)
    for each method:
       loss[method, cell, rep] = evaluate(method, data, truth)
  # Method comparisons within cell are PAIRED across rep
  diff[A vs B, cell] = mean(loss[A, cell, :] - loss[B, cell, :])
  MCSE_diff = sd_paired / sqrt(B)

This is a free variance reduction and is the standard for top-stat-journal method comparisons. Report paired differences with MCSE, not just per-method means.

1F: Conditional diagnostics — REQUIRED when the corresponding claim exists

These are conditional but mandatory once their trigger exists. Top-stat-journal referees will demand them. Skipping = inviting major revision.

1G: Anti-cherry-picking discipline (preregister the headline)

Before running, preregister:

Record these in SIMULATION_PLAN.md BEFORE running anything. If the data later contradict the predesignated headline, REPORT the contradiction — do not silently swap the headline to a more flattering cell.

This is not formal multiplicity correction; it is the discipline that distinguishes "simulation supports my method" from "simulation found cherry-picked cells that support my method." Top-journal referees can spot this from a mile away.

Step 2: Write the Simulation Code

Full operational detail — language choice, code structure, reproducibility tiers, manifest-driven immutable cells, storage format, and edge cases — is in ../stat-shared-references/simulation-execution-protocol.md.

Hard gates for this step:

• STRICT reproducibility is the default for top-stat-journal work: hierarchical RNG streams (not a single global seed), the reproducibility tier declared explicitly, and every cell re-runnable in isolation.
• Manifest-driven, immutable cells. A cell's inputs are fixed once written; a changed design is a new cell, never an edited one. This is what makes partial reruns trustworthy.
• Failure handling is required, not optional. Nonconvergence, singular Hessians, and optimizer stalls are caught, logged per cell, and reported. A silently dropped failure is a referee magnet and biases every metric in that cell.
• Paired replicates. All methods run on the same synthetic data within a replicate, so comparisons are paired rather than independent.

Step 3: Run + Collect Results

3A: Run order

1. Pilots: small B (50), small n grid, all experiments — catch coding bugs
2. Baseline experiments: full B for the main rate-verification figures
3. Stress tests: full B for the assumption-violation figures
4. Sensitivity: vary nuisance params (smoothness, signal strength, etc.)

3B: Result aggregation per cell

For each (n, d, DGP) cell, aggregate B replications into:

• mean, sd, median, IQR of the metric
• Monte Carlo standard error: sd / sqrt(B)
• 95% empirical CI of the metric

Write aggregated to results/aggregated.csv for plotting.

3C: Reproducibility check

Re-run one experiment with the same seed → verify exact match. Document this in README.md.

Step 4: Publication-Grade Figures

General figure conventions — no plot titles, content-bearing captions, legend placement, colour-blind-safe palettes, redundant encoding, multi-panel labelling, vector output, per-venue rules — are owned by stat-figure-design.md (writing repo; resolves in the shared install). Follow it; do not restate it here.

Simulation-specific requirements this skill adds:

• Monte Carlo uncertainty is mandatory. Every plotted point carries an MCSE error bar or a shaded band. A point-estimate plot without uncertainty is not publishable, and this is a hard gate, not a preference.
• Theory reference lines. Show the predicted rate slope (dashed black) and any nominal level such as 0.95 coverage (dashed grey), so the reader can see prediction against evidence in one glance.
• Failure rates are reported, per cell, whenever any cell exceeds 5% nonconvergence.

Figure choice follows the simulation claim:

Runnable matplotlib template and the full convention detail: ../stat-shared-references/examples/simulation-figure-template.md.

Step 4F: Codex Adversarial Review of Simulation Design

Follow ../stat-shared-references/codex-protocol.md. Codex is an adversarial reviewer to discuss with iteratively, not an oracle: every finding gets an explicit ACCEPT / PUSH BACK / REQUEST CLARIFICATION with reasoning. This matters more here than elsewhere, because reflexively accepting a Codex objection can trigger CPU-days of needless reruns.

Two passes: pre-run on the plan (claim coverage, DGP difficulty, n-grid range, replication count per metric, baseline choice, rate-identification protocol) and post-run on the findings (does the evidence actually support each claim).

Emit simulation/codex_discussion.md with the round-by-round record and simulation/codex_design_review.md with the per-finding reconciliation (issue, severity, Codex position, action taken).

Hard rule: when Codex flags a claim OVERCLAIMED or a result SUSPICIOUS, never silently edit RECONCILIATION.md to match. Surface the disagreement to the user and record both positions.

Prompt text and reconciliation-table shape: ../stat-shared-references/examples/simulation-codex-review-prompts.md.

Step 5: Theory ↔ Simulation Reconciliation (HYPOTHESIS GENERATION ONLY)

**CRITICAL DISCIP

…(truncated)