PlanExeOrg Validate napkin_math parameters.json
Overview
A thin wrapper around experiments/napkin_math/validate_parameters.py. The script reads parameters.json and emits validation.json next to it. Output shape is what summarize_assessment.py consumes (named checks_performed list + per-violation rule_id/severity/path/message/suggested_fix).
This replaces the older LLM-driven validate-parameters skill, which was written against an earlier schema and rejected the output_name/output_unit fields the digest extractor is required to emit. The Python validator runs in milliseconds, costs no tokens, and is deterministic.
When to Use
- The extractor (digest or full) has just produced
parameters.jsonand the pipeline needsvalidation.jsonbefore running scenarios or Monte Carlo - The user asks to "validate", "check", or "lint" a parameters file
- After hand-editing
parameters.json, to confirm the result is still structurally valid
Not for: regenerating the parameters (use extract-parameters-from-digest or extract-parameters-from-full), generating bounds, or running calculations.
Workflow
Get the parameters path. If the user did not provide one, ask. Do not guess.
Invoke the script. Requires Python 3.11+ (no extra deps):
/opt/homebrew/bin/python3.11 experiments/napkin_math/validate_parameters.py \ --parameters <path>/parameters.json \ [--output <path>/validation.json]Default output:
<dir-of-parameters>/validation.json. Exit code 0 onvalid: true, 1 on validation errors, 2 on JSON parse failure. The script prints the output path on stdout.Report back. Tell the user the output path, the verdict (
valid/INVALID), and the error count. If there are violations, quote the first fewrule_id+messagepairs verbatim. Don't paraphrase the verdict — the rule ids are how downstream tooling routes the next step.
The 16 checks
| Check | Severity bias | What it checks |
|---|---|---|
json_parse |
ERROR | the file parses as JSON (failure handled with a json_parse violation) |
top_level_structure |
ERROR | plan_summary + four arrays present |
required_fields |
ERROR | each entry carries its required keys |
array_length_caps |
ERROR | ≤8 key_values, ≤5 derived_questions / missing_values_to_estimate / recommended_first_calculations |
global_id_uniqueness |
ERROR | ids unique across all four arrays |
snake_case_ids |
ERROR | ids match ^[a-z][a-z0-9_]*$ |
depends_on_declared |
ERROR | every id in depends_on matches a declared id or output_name |
formula_rhs_declared |
ERROR | every snake_case identifier on the RHS of formula_hint is declared (or is the entry's own output_name); built-ins like min/max are exempt |
fraction_value_range |
ERROR | unit == "fraction" ⇒ value in [0, 1] or null |
comment_word_caps |
ERROR | key_value comment ≤25 words |
source_text_word_caps |
ERROR | key_value source_text ≤20 words |
output_name_present_when_formula_hint |
ERROR | non-empty formula_hint ⇒ output_name not null |
output_unit_present_when_formula_hint |
ERROR | non-empty formula_hint ⇒ output_unit not null |
no_dead_end_variables |
ERROR | every key_value and missing_value is consumed (transitively) by some calculation |
threshold_friendly_naming |
WARN | output_names ending in _gap / _deficit / _shortfall are flagged because they read ambiguously under a >= 0 / <= 0 threshold |
shared_pool_legitimacy |
(no-op) | listed in checks_performed for completeness; enforcement is upstream in the extractor's system prompt (requires reading source narrative to verify pool legitimacy, which is not a structural check) |
valid is true iff error_count == 0. WARN-level findings do not invalidate the file.
Output shape
{
"valid": true,
"error_count": 0,
"warn_count": 0,
"violations": [],
"summary": {
"counts": {"key_values": 8, "derived_questions": 3, "missing_values_to_estimate": 4, "recommended_first_calculations": 5},
"rule_id_breakdown": {},
"checks_performed": ["json_parse", "top_level_structure", ...]
}
}
The summary.checks_performed list is what summarize_assessment.py surfaces as the "Validated" line under ## Confidence and trust boundaries. Keep it as the authoritative list of what the validator actually ran.
Common mistakes
| Mistake | Fix |
|---|---|
Running before parameters.json exists |
Run the extractor first (extract-parameters-from-digest or extract-parameters-from-full). |
| Treating WARN findings as blocking | They are not. valid is set only by ERROR-level findings. |
Editing parameters.json by hand to silence a no_dead_end_variables ERROR |
The right fix is usually to add a calculation that consumes the variable, not to drop the variable. Re-validate after either change. |
Editing parameters.json to silence a threshold_friendly_naming WARN |
If the output is actually threshold-tested, rename and flip the formula sign. If it isn't, the WARN is just advisory. |
Reference
- Script (authoritative):
experiments/napkin_math/validate_parameters.py - Companion skills:
../extract-parameters-from-digest/SKILL.md,../extract-parameters-from-full/SKILL.md,../generate-bounds/SKILL.md,../monte-carlo/SKILL.md,../summarize-assessment/SKILL.md - Example output: any
validation.jsonunderexperiments/napkin_math/output/<version>/<plan>/