# Run Napkin Math Pipeline > Use when the user wants to run the napkin-math pipeline end-to-end on a PlanExe report, or resume a partially populated output directory by filling in only the missing stages. Orchestrates digest preparation, parameter extraction, validation, bounds, calculations, scenarios, Monte Carlo, and assessment rendering. Never copies artifacts forward from prior runs, and never re-runs a stage whose output is already on disk. - Skill: `planexeorg/run-napkin-math-pipeline` (Agent Skill) - Install (CLI): `npx skillmds@latest add planexeorg/run-napkin-math-pipeline` - Raw SKILL.md: https://api.skillmd.com/api/skills/planexeorg/run-napkin-math-pipeline/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: PlanExeOrg (https://skillmd.com/u/planexeorg) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/planexeorg/run-napkin-math-pipeline --- # Run the Napkin-Math Pipeline ## Overview End-to-end orchestrator for the pipeline documented in `experiments/napkin_math/README.md`. Decides which stages to run based on what is already present in the target output directory, then walks forward through the remaining stages one at a time. PlanExe source reports live under `/Users/neoneye/git/PlanExe-web/` (e.g. `/Users/neoneye/git/PlanExe-web/20251114_paperclip_automation/`). Outputs land under `experiments/napkin_math/output///`. ## What the orchestrator does and does not do The orchestrator is **dispatch-only**. It looks at filenames in the target directory, decides which stage is missing, and delegates the work to the appropriate sibling skill (via a subagent) or Python runner. It does **not** read the content of large input files into its own context. **Do not read into orchestrator context:** - `extract_parameters_input.md` (the digest — typically ~25K tokens) - `compress_*.md` or `compress_*_raw.json` files - The raw PlanExe HTML report - The full `parameters.json`, `bounds.json`, `scenarios.json`, `montecarlo.json` (except quick `jq`/Python extraction of a single field) **OK to read into orchestrator context:** - Directory listings (filenames only) - `validation.json` exit status (via Python one-liner, not full file) - `montecarlo.json` per-threshold `probability` values (via Python one-liner) - Sibling skill SKILL.md / system-prompt.txt **only when updating the skill itself** Pulling a digest into orchestrator context defeats the architecture. Each stage's sibling skill knows how to read its own inputs; the orchestrator's job is to invoke that skill, not to do its work. ## Per-stage delegation For each missing stage, dispatch as follows. Pass the agent the file paths and a one-line task; do not paste file contents into the prompt. | Stage | How to dispatch | |---|---| | 0. Digest | `Bash` → `prepare_extract_input.py --planexe-dir --output-dir ` | | 1. Parameters | `Agent` with the sibling skill name `extract-parameters-from-digest`; prompt: "Read `/extract_parameters_input.md` per `system-prompt.txt`, write the result to `/parameters.json`." | | 2. Validation | `Bash` → `validate_parameters.py --parameters … --output …` | | 3. Bounds | `Agent` with `generate-bounds`; prompt: "Read `/parameters.json` per the generate-bounds rules, write `/bounds.json`." | | 4. Calculations | `Agent` with `generate-calculations`; prompt: "Read `/parameters.json`, write `/calculations.py` per the skill rules." | | 5. Scenarios | `Agent` with `run-scenarios`; prompt: "Read parameters.json, bounds.json, calculations.py, write `/scenarios.json`." | | 6. MC settings | Small hand-written JSON file based on `parameters.json` — extract `output_name`s via Python one-liner, write thresholds with `>= 0`. No content reading needed. | | 7. Monte Carlo | `Bash` → `run_monte_carlo.py …` | | 8. Assessment | `Bash` → `summarize_assessment.py …` | After each agent returns, verify the output file exists and is non-empty, then move on. If validation fails after Stage 1, dispatch a repair agent with the validation report — do not pull the parameters into context yourself. ## Why resume mode exists Resume is not a convenience. It is the experimental control for this pipeline. Several stages — digest preparation, parameter extraction, bounds, calculations, scenarios — are LLM-driven and therefore non-deterministic. If every run regenerated them from scratch, every downstream change would be confounded with upstream sampling noise, and the user could not tell whether a different `assessment.md` came from a skill/prompt tweak or just from a different roll of the LLM. The intended workflow is: 1. On the first run for a plan, populate the directory once. The `extract_parameters_input.md` digest (Stage 0) becomes the pinned starting point. 2. On subsequent runs — typically after editing a skill prompt, a validator rule, or a runner — the user deletes only the artifacts downstream of the change, and resumes from the highest-numbered stage whose output is still present. 3. Because the input to the first re-run stage is byte-identical to the prior run, any downstream difference can be attributed to the tweak under test. The corollary is strict: **never re-run a stage whose output is already on disk.** Doing so silently changes the input that every later stage sees, and destroys the comparability the user is trying to preserve. If the user wants a stage re-run, they will delete its output file first; absence of the file is the signal to run, presence is the signal to leave alone. ## The two cardinal rules **1. Never copy a pipeline artifact from a different output directory into the target directory.** Skill prompts, validator rules, and runner logic evolve between versions; copying `output///` into `output///` silently embeds the old rules into the new run. If at any point you are about to type `cp /…/v{N-1}// $D/`, stop and run the stage that produces `` instead. **2. Never re-run a stage whose output is already on disk.** The user's A/B comparability depends on the input to each stage being byte-identical to the prior run, up to the first deleted artifact. Re-running a present stage (especially Stage 0, the LLM-driven digest) re-rolls the dice on its input distribution and confounds every downstream comparison. The only file presence the orchestrator changes is by *adding* missing files, never by *replacing* present ones. Together: file present ⇒ leave alone; file absent ⇒ produce it from this stage's runner (never by copy). ## Pipeline file map Each stage produces exactly one or two files in the target directory. A stage is "done" iff its output file(s) exist in the target dir. | Order | Stage | Producer | Output file(s) | |---|---|---|---| | 0 | Digest preparation | `prepare_extract_input.py` (Python) | `compress_*.md`, `compress_*_raw.json` (4 sections), `extract_parameters_input.md` | | 1 | Parameter extraction | `extract-parameters-from-digest` skill (LLM) | `parameters.json` | | 2 | Validation | `validate_parameters.py` (Python) | `validation.json` | | 3 | Bounds | `generate-bounds` skill (LLM) | `bounds.json` | | 4 | Calculations | `generate-calculations` skill (LLM) | `calculations.py` | | 5 | Scenarios | `run-scenarios` skill (LLM, deterministic) | `scenarios.json` | | 6 | Monte Carlo settings | hand-written | `montecarlo_settings.json` | | 7 | Monte Carlo | `run_monte_carlo.py` (Python) | `montecarlo.json` | | 8 | Assessment | `summarize_assessment.py` (Python) | `assessment.md` | `montecarlo_settings.json` is a tiny JSON file (n_runs, seed, distribution_default, thresholds) — write it from the threshold list in `recommended_first_calculations`, with `>= 0` operator for every margin/surplus/buffer output. Standard values: `n_runs: 10000`, `seed: 12345`, `distribution_default: "triangular"`. ## Inputs the orchestrator needs Two scenarios: **(a) Fresh start from a PlanExe-web report.** User points at `/Users/neoneye/git/PlanExe-web/_/` and a target version. Create `output///` if it doesn't exist. Run stage 0 first. **(b) Resume from a partially populated output directory.** User points at `output///`. List the dir, classify which stages are done, run forward from the first missing stage. If the user did not name a version, ask. Do not guess from sibling directories (last-mtime, etc.). ## Workflow 1. **Identify the target directory.** Absolute path under `experiments/napkin_math/output///`. 2. **List its contents.** Decide per stage whether the output file is present. 3. **Find the first missing stage.** Run it. After it completes, re-check and continue with the next missing stage. Do not batch multiple stages together until you have verified the previous one landed. 4. **For LLM-driven stages, invoke the corresponding sibling skill.** Each sibling skill's `system-prompt.txt` is authoritative — read it and apply its rules. The orchestrator does not paraphrase or short-circuit those rules. 5. **For Python stages, invoke the script.** See the commands below. 6. **Re-validate after each non-deterministic stage.** Specifically: after parameter extraction, run validate-parameters and confirm `valid: true` before moving on. If `valid: false`, hand-fix the reported violations and re-validate; do not proceed with invalid parameters. 7. **Final summary.** When `assessment.md` exists and Monte Carlo completed with zero warnings, report the worst-gate verdict, the five pass probabilities, and the overall band. ## Commands (Python stages) Always invoke Python 3.11 explicitly (the runner uses NumPy + llama_index that resolve under that interpreter): ```sh PY=/opt/homebrew/bin/python3.11 NM=/Users/neoneye/git/PlanExeGroup/PlanExe/experiments/napkin_math ``` Stage 0 — digest preparation (creates the 8 compress files + the digest): ```sh $PY $NM/prepare_extract_input.py \ --planexe-dir /Users/neoneye/git/PlanExe-web/_ \ --output-dir $NM/output// ``` Stage 2 — validation: ```sh $PY $NM/validate_parameters.py \ --parameters $D/parameters.json \ --output $D/validation.json ``` Stage 7 — Monte Carlo: ```sh $PY $NM/run_monte_carlo.py \ --parameters $D/parameters.json \ --bounds $D/bounds.json \ --calculations $D/calculations.py \ --settings $D/montecarlo_settings.json \ --output $D/montecarlo.json ``` Stage 8 — assessment rendering: ```sh $PY $NM/summarize_assessment.py \ --parameters $D/parameters.json \ --bounds $D/bounds.json \ --scenarios $D/scenarios.json \ --montecarlo $D/montecarlo.json \ --validation $D/validation.json \ --settings $D/montecarlo_settings.json \ --extract-input $D/extract_parameters_input.md \ --calculations $D/calculations.py \ --output $D/assessment.md ``` After Stage 7, delete the `__pycache__/` that the runner creates next to `calculations.py`: ```sh rm -rf $D/__pycache__ ``` ## LLM-driven stages — how to actually run them For stages 1, 3, 4, 5: do not paste the upstream artifacts and improvise. Walk through the sibling skill's workflow: - Stage 1 — `extract-parameters-from-digest` - Read `$D/extract_parameters_input.md`. - Read `../extract-parameters-from-digest/system-prompt.txt`. - Produce `parameters.json` per the schema at the end of the system prompt. Apply every "Important", "Additional modelling rules", and "Formula and dependency rules" section. - Stage 3 — `generate-bounds` - Read `$D/parameters.json` (must be valid). - Read `../generate-bounds/system-prompt.txt` and SKILL.md. - Apply the actual-vs-commitment default: base centers at the source-named committed value unless a named Risk / Issue / Decision / premortem / expert-criticism passage forecasts a gap. - Skip threshold/target variables (ids ending in `_threshold`, `_target`, `_ceiling`, `_floor`, `_limit`) — they enter the simulation as their stated single value. - Stage 4 — `generate-calculations` - One Python function per entry in `recommended_first_calculations`. Function name = `output_name`; arguments = `depends_on`. No hidden state; add divide-by-zero guards only where the formula divides. - Stage 5 — `run-scenarios` - Build three input pools (low / base / high) by selecting the matching value from `bounds.json` for bounded variables, and the declared `key_values[*].value` for unbounded variables. - Run each `recommended_first_calculations` function against each pool. - Emit `scenarios.json` with `inputs`, `outputs`, `comparison` (low/base/high + spread_ratio + spread_absolute), and any `warnings`. Compute spread arithmetically; do not copy the structure from a prior run. ## Resume-mode decision table When the user points at a partially populated dir, decide by file presence only — never by sibling-directory comparison. | Present | First missing | Action | |---|---|---| | nothing | digest | If user gave a PlanExe-web dir, run Stage 0. If they only gave the output dir, ask for the source dir. | | 8 compress files + digest | `parameters.json` | Run Stage 1. | | + `parameters.json` | `validation.json` | Run Stage 2. If validation reports errors, fix the parameters and re-validate before continuing. | | + `validation.json` (valid) | `bounds.json` | Run Stage 3. | | + `bounds.json` | `calculations.py` | Run Stage 4. | | + `calculations.py` | `scenarios.json` | Run Stage 5. | | + `scenarios.json` | `montecarlo_settings.json` | Write settings (n_runs 10000, seed 12345, triangular, `>= 0` thresholds per recommended_first_calculations). | | + `montecarlo_settings.json` | `montecarlo.json` | Run Stage 7. Expect 0 warnings; if not, diagnose. | | + `montecarlo.json` | `assessment.md` | Run Stage 8. | If an intermediate file is present but a later one is also present with stale numbers (e.g. `bounds.json` updated but `montecarlo.json` not regenerated), ask the user whether to keep or regenerate the later file. Do not silently overwrite. ## Anti-patterns | Pattern | What's wrong | Do this instead | |---|---|---| | `cp $D47/parameters.json $D49/` | Bypasses Stage 1 entirely; embeds stale extractor rules | Read the digest and produce a fresh `parameters.json` per the current extract-parameters-from-digest system prompt | | `cp $D48/bounds.json $D49/` and claiming a "fresh run" | The generate-bounds skill's actual-vs-commitment rule and threshold-skip rule have changed multiple times; copying defeats the point of the update | Read `parameters.json`, apply the current generate-bounds rules from scratch, justify each `low/base/high` against a named source passage | | `diff -q ...` finds digests are identical, so copy everything forward | Identical digest means Stage 0 doesn't need to re-run. It does NOT mean Stages 1–8 can be copied — those use the current LLM-driven skills, which may have evolved | Re-run Stages 1–8 freshly. If the bounds genuinely land in the same place, that's a valid outcome — but you have to do the work to know | | Re-running Stage 0 (digest) when `extract_parameters_input.md` is already present, "to be safe" | Destroys the experimental control. The whole point of resuming is to pin the digest so downstream changes are attributable to the tweak under test. Re-rolling it confounds the comparison silently | If the user wants the digest regenerated, they will delete it first. Presence of the file is an explicit instruction to leave it alone | | Re-running any present LLM-driven stage to "freshen" it | Same as above — re-roll changes the input that the next stage sees, defeating A/B comparability | Only run stages whose output file is absent | | Skip validation because "parameters look fine" | Validator catches comment word-cap, dead-end variables, formula RHS not declared, fraction range, and more | Always run `validate_parameters.py` after extraction and confirm `valid: true` | | Write `montecarlo_settings.json` with thresholds copied from another plan | Threshold ids must match this plan's `recommended_first_calculations` `output_name`s | Build the thresholds map from the current `parameters.json` | | Report the verdict after a stage emits warnings | Warnings often signal threshold-stripping or missing dependencies that change the meaning of pass probabilities | Resolve warnings first, then report | | Hand-compute Monte Carlo pass probabilities | The LLM can't sample 10k draws correctly in-prompt | Always invoke `run_monte_carlo.py` | ## Self-audit before final report Before declaring the run complete, verify: 1. `validation.json` shows `valid: true` and 0 errors. 2. `montecarlo.json` has 0 warnings, or the warnings are understood and acknowledged in the summary. 3. Every threshold id in `montecarlo_settings.json` matches an `output_name` in `parameters.json`'s `recommended_first_calculations`. 4. `assessment.md` exists and its `## Machine summary` block lists `overall_risk_band` consistent with the worst per-gate pass probability under the band table: | Pass rate | Band | |---|---| | ≥ 0.80 | Robust | | 0.50 – 0.80 | Marginal | | 0.20 – 0.50 | Fragile | | < 0.20 | Critical | 5. No `__pycache__/` directory remains in the output dir. If any check fails, fix it before reporting back. Reporting "done" with a known issue is worse than asking the user how to proceed. ## Reference - Pipeline overview: `../../../README.md` (Stages 1–7 + design principles) - Sibling skill prompts (authoritative for each stage): `../extract-parameters-from-digest/`, `../generate-bounds/`, `../generate-calculations/`, `../run-scenarios/`, `../monte-carlo/`, `../summarize-assessment/`, `../validate-parameters/` - Python runners: `../../../prepare_extract_input.py`, `../../../validate_parameters.py`, `../../../run_monte_carlo.py`, `../../../summarize_assessment.py` - PlanExe source reports: `/Users/neoneye/git/PlanExe-web/_/` - Output convention: `experiments/napkin_math/output/v//`