Fit Pipeline
Every experiment's stages must leave behind the same files with the same
layouts: develop.js audits them, the selector ranks on them, and report-quant
sources numbers from them. This skill is that contract plus reference code that
honors it. There is no library to import. Copy the script you need into the
stage directory, adapt the marked section, run it with uv run from the
project root, and keep it — *.py is an artifact.
Reference scripts
| Script |
Used by |
Writes |
references/posterior_fit.py |
model-fitter (probe and full run); recovery fits |
summary.json, diagnostics.json, loo.json, thinned_draws.npz, posterior.nc |
references/prior_predictive.py |
prior-predictive-checker |
prior_predictive.nc; prior_check.json when --bounds given |
references/fake_data.py |
fake-data-checker (simulate, then check after a recovery fit) |
fake_data.json, true_params.json, recovery.json |
CLI usage is at the top of each file. posterior_fit.py is the only one with a
part you are expected to rewrite — build_stan_data(), which maps the dispatched
dataset onto the model's data {} block. Stan-JSON input passes through
unchanged, which is how the recovery fit consumes fake_data.json.
Artifact contract
posterior_fit.py writes, under the stage directory:
| File |
Layout |
summary.json |
model_name; param_summary (az.summary columns mean, sd, hdi_3%, hdi_97%, ess_bulk, ess_tail, r_hat, keyed column → parameter); convergence; diagnostics; loo (absent without log_lik); artifacts (absolute path of every file of the save, itself included); warnings |
diagnostics.json |
num_divergences, max_treedepth_exceeded, ebfmi_warnings — CmdStanPy's own counters plus ArviZ E-BFMI |
loo.json |
elpd_loo, se, p_loo, k_good (k < 0.5), k_ok (< 0.7), k_bad (< 1), k_very_bad (≥ 1) |
thinned_draws.npz |
200 parameter-only draws per variable, sample axis first — never y_rep or log_lik |
posterior.nc |
InferenceData with posterior, posterior_predictive (y_rep), log_likelihood (log_lik), sample_stats, observed_data (y) |
convergence is {max_rhat, min_ess_bulk, min_ess_tail, n_divergent, converged}
where converged applies the thresholds below. prior_check.json carries
extreme_draw_pct against the assigned bounds; recovery.json carries
coverage_90 and max_bias_z with a per-parameter breakdown.
Audited fields (validation-protocol > Audited numeric fields) map onto these
files directly: rhat_max ← convergence.max_rhat; ess_min ←
min(min_ess_bulk, min_ess_tail); divergences ← n_divergent; score,
score_se ← loo.elpd_loo, loo.se when the plan's metric is observation-level
LOO; pareto_k_bad_pct ← 100 · (k_bad + k_very_bad) / N; extreme_draw_pct,
coverage_90, max_bias_z ← the prior and recovery JSON files.
Rules the scripts encode — keep them when you adapt
- Quiet sampling.
show_progress=False, show_console=False, always;
leave refresh at its default (CmdStanPy ≥ 1.3 rejects refresh=0).
Progress output floods the agent transcript and can crash the session.
- NUTS needs warmup > 0. GQ-only programs run with
fixed_param=True, iter_warmup=0, adapt_engaged=False and no adapt_delta — CmdStanPy
rejects any adapt_* setting once adaptation is off.
- Never bare
az.from_cmdstanpy(fit). Name posterior_predictive,
log_likelihood, and observed_data explicitly, and only pass variables the
program declares (fit.metadata.stan_vars), or conversion crashes.
- Convergence from
az.summary, not fit.diagnose(), which OOMs above
~10K observations. Thresholds: R̂ < 1.01, ESS ≥ 400 bulk and tail, 0
divergences, E-BFMI ≥ 0.3 (ref: convergence-diagnostics).
- Delete the CmdStan CSVs once draws are in memory. They are 5–500 MB per
chain and nothing in them is lost after conversion.
- JSON through
NumpyEncoder. numpy scalars are not serializable.
- Subsample N > 2000 before prior simulation (ref:
stan > GQ-only pitfalls).
- Non-LOO ranking metrics (grouped, leave-future-out) are computed in your
own script from
posterior.nc's log_likelihood; mirror loo() in
posterior_fit.py for the observation-level shape and write
ranking_score.json as the model-fitter agent describes.
Tests
The reference scripts are executed in CI against real CmdStan (repository
tests/): the artifact contract, parameter recovery, divergence flagging, LOO
ranking, the prior and recovery numbers, and each CLI are asserted. If you change
a layout here, change the tests.
1---2name: fit-pipeline3description: How a Stan fit becomes the canonical artifacts the workflow audits — sampling defaults, InferenceData construction, convergence and LOO numbers, the file contract — with three runnable reference scripts (posterior fit, prior predictive, fake-data simulation + recovery) to copy and adapt.4---56# Fit Pipeline78Every experiment's stages must leave behind the same files with the same9layouts: `develop.js` audits them, the selector ranks on them, and report-quant10sources numbers from them. This skill is that contract plus reference code that11honors it. **There is no library to import.** Copy the script you need into the12stage directory, adapt the marked section, run it with `uv run` from the13project root, and keep it — `*.py` is an artifact.1415## Reference scripts1617| Script | Used by | Writes |18|---|---|---|19| `references/posterior_fit.py` | model-fitter (probe and full run); recovery fits | `summary.json`, `diagnostics.json`, `loo.json`, `thinned_draws.npz`, `posterior.nc` |20| `references/prior_predictive.py` | prior-predictive-checker | `prior_predictive.nc`; `prior_check.json` when `--bounds` given |21| `references/fake_data.py` | fake-data-checker (`simulate`, then `check` after a recovery fit) | `fake_data.json`, `true_params.json`, `recovery.json` |2223CLI usage is at the top of each file. `posterior_fit.py` is the only one with a24part you are expected to rewrite — `build_stan_data()`, which maps the dispatched25dataset onto the model's `data {}` block. Stan-JSON input passes through26unchanged, which is how the recovery fit consumes `fake_data.json`.2728## Artifact contract2930`posterior_fit.py` writes, under the stage directory:3132| File | Layout |33|---|---|34| `summary.json` | `model_name`; `param_summary` (`az.summary` columns `mean`, `sd`, `hdi_3%`, `hdi_97%`, `ess_bulk`, `ess_tail`, `r_hat`, keyed column → parameter); `convergence`; `diagnostics`; `loo` (absent without `log_lik`); `artifacts` (absolute path of every file of the save, itself included); `warnings` |35| `diagnostics.json` | `num_divergences`, `max_treedepth_exceeded`, `ebfmi_warnings` — CmdStanPy's own counters plus ArviZ E-BFMI |36| `loo.json` | `elpd_loo`, `se`, `p_loo`, `k_good` (k < 0.5), `k_ok` (< 0.7), `k_bad` (< 1), `k_very_bad` (≥ 1) |37| `thinned_draws.npz` | 200 parameter-only draws per variable, sample axis first — never `y_rep` or `log_lik` |38| `posterior.nc` | InferenceData with `posterior`, `posterior_predictive` (`y_rep`), `log_likelihood` (`log_lik`), `sample_stats`, `observed_data` (`y`) |3940`convergence` is `{max_rhat, min_ess_bulk, min_ess_tail, n_divergent, converged}`41where `converged` applies the thresholds below. `prior_check.json` carries42`extreme_draw_pct` against the assigned bounds; `recovery.json` carries43`coverage_90` and `max_bias_z` with a per-parameter breakdown.4445**Audited fields** (`validation-protocol > Audited numeric fields`) map onto these46files directly: `rhat_max` ← `convergence.max_rhat`; `ess_min` ←47`min(min_ess_bulk, min_ess_tail)`; `divergences` ← `n_divergent`; `score`,48`score_se` ← `loo.elpd_loo`, `loo.se` when the plan's metric is observation-level49LOO; `pareto_k_bad_pct` ← `100 · (k_bad + k_very_bad) / N`; `extreme_draw_pct`,50`coverage_90`, `max_bias_z` ← the prior and recovery JSON files.5152## Rules the scripts encode — keep them when you adapt5354- **Quiet sampling.** `show_progress=False, show_console=False`, always;55 leave `refresh` at its default (CmdStanPy ≥ 1.3 rejects `refresh=0`).56 Progress output floods the agent transcript and can crash the session.57- **NUTS needs warmup > 0.** GQ-only programs run with `fixed_param=True,58 iter_warmup=0, adapt_engaged=False` and **no `adapt_delta`** — CmdStanPy59 rejects any `adapt_*` setting once adaptation is off.60- **Never bare `az.from_cmdstanpy(fit)`.** Name `posterior_predictive`,61 `log_likelihood`, and `observed_data` explicitly, and only pass variables the62 program declares (`fit.metadata.stan_vars`), or conversion crashes.63- **Convergence from `az.summary`, not `fit.diagnose()`**, which OOMs above64 ~10K observations. Thresholds: R̂ < 1.01, ESS ≥ 400 bulk and tail, 065 divergences, E-BFMI ≥ 0.3 (ref: `convergence-diagnostics`).66- **Delete the CmdStan CSVs** once draws are in memory. They are 5–500 MB per67 chain and nothing in them is lost after conversion.68- **JSON through `NumpyEncoder`.** numpy scalars are not serializable.69- **Subsample N > 2000** before prior simulation (ref: `stan > GQ-only pitfalls`).70- **Non-LOO ranking metrics** (grouped, leave-future-out) are computed in your71 own script from `posterior.nc`'s `log_likelihood`; mirror `loo()` in72 `posterior_fit.py` for the observation-level shape and write73 `ranking_score.json` as the model-fitter agent describes.7475## Tests7677The reference scripts are executed in CI against real CmdStan (repository78`tests/`): the artifact contract, parameter recovery, divergence flagging, LOO79ranking, the prior and recovery numbers, and each CLI are asserted. If you change80a layout here, change the tests.
sunxd3