Error Budget Experimental Design Lens
Philosophical Mode: Statistical
Primary Question: "Are error risks sized and controlled?"
Focus: Type I/II Errors, Power, Minimum Detectable Effect, Multiplicity, Sequential Monitoring
Arguments
/autoskillit:exp-lens-error-budget [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Need to verify statistical power before running an experiment
- Multiple comparisons are performed without a stated correction strategy
- Sequential testing or interim analysis is in use without defined stopping rules
- User invokes
/autoskillit:exp-lens-error-budget or /autoskillit:make-experiment-diag error
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Accept default alpha=0.05 without checking whether it is appropriate for the decision context
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-error-budget/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Enumerate every statistical test and account for its error contribution
Distinguish per-test error rates from family-wise error rates
Flag any sequential peeking without a formal stopping rule as a critical defect
Evaluate whether the minimum detectable effect is practically meaningful, not just statistically chosen
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-error-budget/exp_diag_error_budget_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Sample Size & Power
- Find power calculations or sample size justifications
- Look for: power, sample_size, n_samples, effect_size, minimum_detectable, mde
Multiple Comparisons
- Find all statistical tests performed and correction strategies
- Look for: bonferroni, fdr, holm, bh, correction, multiple, comparisons, tests
Sequential Analysis
- Find interim analyses, stopping rules, or sequential monitoring
- Look for: interim, early_stopping, sequential, alpha_spending, peek, monitor
Decision Thresholds
- Find significance thresholds and decision rules
- Look for: alpha, p_value, threshold, significance, reject, null, hypothesis
Effect Size Context
- Find practical significance alongside statistical significance
- Look for: effect_size, cohen, practical, meaningful, magnitude, difference
Step 2: Build the Error Budget
For each statistical claim:
- What is the per-test Type I error rate?
- What is the family-wise Type I error rate?
- What is the power (1 - Type II error)?
- What is the minimum detectable effect?
- Is sequential monitoring in use, and if so, what stopping rule is defined?
- Is the chosen alpha appropriate for the decision context?
Step 3: Analyze Error Allocation
For each test, rate alignment as: ALIGNED / CONVENTIONAL / MISALIGNED
Step 4: Create Optional Decision-Flow Diagram
If a diagram adds value, show Data → Tests → Thresholds → Conclusions, with labeled error rates.
Step 5: Write Output
Write the analysis to: {{AUTOSKILLIT_TEMP}}/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Pre-Diagram Checklist
Before creating the diagram, verify:
Related Skills
/autoskillit:make-experiment-diag - Parent skill
/autoskillit:mermaid - MUST BE LOADED before creating diagram
/autoskillit:exp-lens-severity-testing
/autoskillit:exp-lens-variance-stability
1---2name: exp-lens-error-budget3categories: [exp-lens]4activate_deps: [mermaid]5description: Analyze statistical error budget showing Type I/II errors, power, minimum detectable effect, multiplicity corrections, and sequential monitoring. Statistical lens answering "Are error risks sized and controlled?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Error Budget Lens - Analyzing statistical error risks...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-error-budget15---1617# Error Budget Experimental Design Lens1819**Philosophical Mode:** Statistical20**Primary Question:** "Are error risks sized and controlled?"21**Focus:** Type I/II Errors, Power, Minimum Detectable Effect, Multiplicity, Sequential Monitoring2223## Arguments2425`/autoskillit:exp-lens-error-budget [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- Need to verify statistical power before running an experiment38- Multiple comparisons are performed without a stated correction strategy39- Sequential testing or interim analysis is in use without defined stopping rules40- User invokes `/autoskillit:exp-lens-error-budget` or `/autoskillit:make-experiment-diag error`4142## Critical Constraints4344**NEVER:**45- Modify any source code files46- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves47- Accept default alpha=0.05 without checking whether it is appropriate for the decision context48- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-error-budget/`49- Run subagents in the background (`run_in_background: true` is prohibited)5051**ALWAYS:**52- Enumerate every statistical test and account for its error contribution53- Distinguish per-test error rates from family-wise error rates54- Flag any sequential peeking without a formal stopping rule as a critical defect55- Evaluate whether the minimum detectable effect is practically meaningful, not just statistically chosen56- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY57- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.58- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md`59- After writing the file, emit the structured output token as **literal plain text** with no60 markdown formatting on the token name (the adjudicator performs a regex match):6162 ```63 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-error-budget/exp_diag_error_budget_{...}.md64 ```6566---6768## Analysis Workflow6970### Step 0: Parse optional arguments7172If positional arg 1 (context_path) is provided and the file exists, read it to obtain73IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional74arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full75methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD76exploration for these fields if the context file supplies them.7778### Step 1: Launch Parallel Exploration Subagents7980Spawn Explore subagents to investigate:8182**Sample Size & Power**83- Find power calculations or sample size justifications84- Look for: power, sample_size, n_samples, effect_size, minimum_detectable, mde8586**Multiple Comparisons**87- Find all statistical tests performed and correction strategies88- Look for: bonferroni, fdr, holm, bh, correction, multiple, comparisons, tests8990**Sequential Analysis**91- Find interim analyses, stopping rules, or sequential monitoring92- Look for: interim, early_stopping, sequential, alpha_spending, peek, monitor9394**Decision Thresholds**95- Find significance thresholds and decision rules96- Look for: alpha, p_value, threshold, significance, reject, null, hypothesis9798**Effect Size Context**99- Find practical significance alongside statistical significance100- Look for: effect_size, cohen, practical, meaningful, magnitude, difference101102### Step 2: Build the Error Budget103104For each statistical claim:1051. What is the per-test Type I error rate?1062. What is the family-wise Type I error rate?1073. What is the power (1 - Type II error)?1084. What is the minimum detectable effect?1095. Is sequential monitoring in use, and if so, what stopping rule is defined?1106. Is the chosen alpha appropriate for the decision context?111112### Step 3: Analyze Error Allocation113114For each test, rate alignment as: ALIGNED / CONVENTIONAL / MISALIGNED115116### Step 4: Create Optional Decision-Flow Diagram117118If a diagram adds value, show Data → Tests → Thresholds → Conclusions, with labeled error rates.119120### Step 5: Write Output121122Write the analysis to: `{{AUTOSKILLIT_TEMP}}/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)123124---125126## Pre-Diagram Checklist127128Before creating the diagram, verify:129130- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool131- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)132- [ ] Diagram will include a color legend table133134---135136## Related Skills137138- `/autoskillit:make-experiment-diag` - Parent skill139- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram140- `/autoskillit:exp-lens-severity-testing`141- `/autoskillit:exp-lens-variance-stability`